Hierarchical point cloud compression

ABSTRACT

A system comprises an encoder configured to compress attribute information for a point cloud and/or a decoder configured to decompress compressed attribute information for the point cloud. Attribute values for at least one starting point are included in a compressed attribute information file and attribute correction values used to correct predicted attribute values are included in the compressed attribute information file. Attribute values are predicted based, at least in part, on attribute values of neighboring points and distances between a particular point for whom an attribute value is being predicted and the neighboring points. The predicted attribute values are compared to attribute values of a point cloud prior to compression to determine attribute correction values. A decoder follows a similar prediction process as an encoder and corrects predicted values using attribute correction values included in a compressed attribute information file.

This application is a continuation of U.S. patent application Ser. No.16/133,674, filed Sep. 17, 2018, which is a continuation-in-part of U.S.application Ser. No. 16/130,949, filed Sep. 13, 2018, which claimsbenefit of priority to the following U.S. Provisional Applications:

-   -   U.S. Provisional Application Ser. No. 62/558,795, entitled        “Point Cloud Compression,” filed Sep. 14, 2017;    -   U.S. Provisional Application Ser. No. 62/560,164, entitled        “Hierarchical Point Cloud Compression,” filed Sep. 18, 2017;    -   U.S. Provisional Application Ser. No. 62/569,602, entitled        “Hierarchical Point Cloud Compression,” filed Oct. 8, 2017;    -   U.S. Provisional Application Ser. No. 62/655,759, entitled        “Adaptive Distance Based Point Cloud Compression,” filed Apr.        10, 2018;    -   U.S. Provisional Application Ser. No. 62/655,764, entitled        “Hierarchical Point Cloud Compression with Smoothing,” filed        Apr. 10, 2018;    -   U.S. Provisional Application Ser. No. 62/655,768, entitled        “Point Cloud Attribute Transfer Algorithm,” filed Apr. 10, 2018;        and    -   U.S. Provisional Application Ser. No. 62/696,295, entitled        “Hierarchical Point Cloud Compression,” filed Jul. 10, 2018.        This application incorporates by reference the parent        application (U.S. application Ser. No. 16/133,674), the        continuation-in-part grandparent application (U.S. application        Ser. No. 16/130,949) and each of the above referenced        provisional applications to which the continuation-in-part        grandparent application claims priority, in their entirety.

Additionally, the parent Application (U.S. patent application Ser. No.16/133,674, filed Sep. 7, 2018) of which this Application is acontinuation, also directly claims benefit of priority to the followingU.S. Provisional Applications:

-   -   U.S. Application Ser. No. 62/560,164, entitled “Hierarchical        Point Cloud Compression,” filed Sep. 18, 2017;    -   U.S. Provisional Application Ser. No. 62/569,602, entitled        “Hierarchical Point Cloud Compression,” filed Oct. 8, 2017;    -   U.S. Provisional Application Ser. No. 62/655,759, entitled        “Adaptive Distance Based Point Cloud Compression,” filed Apr.        10, 2018;    -   U.S. Provisional Application Ser. No. 62/655,764, entitled        “Hierarchical Point Cloud Compression with Smoothing,” filed        Apr. 10, 2018;    -   U.S. Provisional Application Ser. No. 62/655,768, entitled        “Point Cloud Attribute Transfer Algorithm,” filed Apr. 10, 2018;    -   U.S. Provisional Application Ser. No. 62/689,021, entitled        “Point Cloud Geometry Compression Using Octrees and Binary        Arithmetic Encoding with Adaptive Look-Up Tables,” filed Jun.        22, 2018;    -   U.S. Provisional Application Ser. No. 62/696,295, entitled        “Hierarchical Point Cloud Compression,” filed Jul. 10, 2018.        This application also incorporates by reference each of the        above referenced provisional applications, to which the parent        application directly claims priority, in their entirety.

BACKGROUND Technical Field

This disclosure relates generally to compression and decompression ofpoint clouds comprising a plurality of points, each having associatedattribute information.

Description of the Related Art

Various types of sensors, such as light detection and ranging (LIDAR)systems, 3-D-cameras, 3-D scanners, etc. may capture data indicatingpositions of points in three dimensional space, for example positions inthe X, Y, and Z planes. Also, such systems may further capture attributeinformation in addition to spatial information for the respectivepoints, such as color information (e.g. RGB values), intensityattributes, reflectivity attributes, motion related attributes, modalityattributes, or various other attributes. In some circumstances,additional attributes may be assigned to the respective points, such asa time-stamp when the point was captured. Points captured by suchsensors may make up a “point cloud” comprising a set of points eachhaving associated spatial information and one or more associatedattributes. In some circumstances, a point cloud may include thousandsof points, hundreds of thousands of points, millions of points, or evenmore points. Also, in some circumstances, point clouds may be generated,for example in software, as opposed to being captured by one or moresensors. In either case, such point clouds may include large amounts ofdata and may be costly and time-consuming to store and transmit.

SUMMARY OF EMBODIMENTS

In some embodiments, a system includes one or more sensors configured tocapture points that collectively make up a point cloud, wherein each ofthe points comprises spatial information identifying a spatial locationof the respective point and attribute information defining one or moreattributes associated with the respective point. The system also includean encoder configured to compress the attribute information for thepoints. To compress the attribute information, the encoder is configuredto assign an attribute value to at least one point of the point cloudbased on the attribute information included in the captured point cloud.Additionally, the encoder is configured to, for each of respective otherones of the points of the point cloud, identify a set of neighboringpoints, determine a predicted attribute value for the respective pointbased, at least in part, on predicted or assigned attributes values forthe neighboring points, and determine, based, at least in part, oncomparing the predicted attribute value for the respective point to theattribute information for the point included in the captured pointcloud, an attribute correction value for the point. The encoder isfurther configured to encode the compressed attribute information forthe point cloud, wherein the compressed attribute information comprisesthe assigned attribute value for the at least one point and dataindicating, for the respective other ones of the points, the respectivedetermined attribute correction values.

In some embodiments, a method for compressing attribute information fora point cloud includes assigning an attribute value to at least onepoint of the point cloud based, at least in part, on attributeinformation for the at least one point included in the point cloud,wherein the point cloud comprises spatial information for a plurality ofpoints and attribute information specifying one or more attributes forrespective ones of the plurality of points. The method further includes,for each of respective other ones of the points of the point cloud,identifying a set of neighboring points and determining a predictedattribute value for the point based, at least in part, on predicted orassigned attribute values for the neighboring points. The method furtherincludes, for each of respective other ones of the points of the pointcloud, determining, based, at least in part, on comparing the predictedattribute value for the point to the attribute information for thepoint, an attribute correction value for the point. The method alsoincludes encoding compressed attribute information for the point cloudcomprising the assigned attribute value for the at least one point anddata indicating, for the respective other ones of the points, thedetermined attribute correction values.

In some embodiments, one or more non-transitory computer-readable mediastore program instructions that, when executed by one or moreprocessors, cause the one or more processors to implement a decoderconfigured to: receive compressed attribute information for a pointcloud comprising at least one assigned attribute value for at least onepoint of the point cloud and data indicating, for other points of thepoint cloud, respective attribute correction values for respectiveattributes of the other points. The program instructions, when executed,further cause the decoder to, for each of respective other ones of thepoints of the point cloud other than the at least one point, identify aset of neighboring points to a point being evaluated, determine apredicted attribute value for the point being evaluated based, at leastin part, on predicted or assigned attribute values for the neighboringpoints, and adjust the predicted attribute value for the point beingevaluated based, at least in part, on an attribute correction value forthe point included in the compressed attribute information. The programinstructions, when executed, further cause the decoder to provideattribute information for a decompressed point cloud comprising the atleast one assigned attribute value for the at least one point and theadjusted predicted attribute values for the other ones of the points.

In some embodiments, to compress attribute information, an encoder isconfigured to build a hierarchical level of detail (LOD) structure. Forexample, the encoder may be configured to determine a first level ofdetail for attribute information of a point cloud and determine one ormore additional levels of detail for the attribute information of thepoint cloud. To determine the first level of detail or the one or moreadditional levels of detail, the encoder is configured to assign anattribute value to at least one point of the point cloud based on theattribute information included in the captured point cloud for the pointand, for a sub-set of respective ones of the points of the point cloudnot included in a previously determined level of detail: identify a setof neighboring points greater than a first distance from the point,determine a predicted attribute value for the respective point based onpredicted or assigned attributes values for the neighboring points, anddetermine, based on comparing the predicted attribute value for therespective point to the attribute information for the point included inthe captured point cloud, an attribute correction value for the point.The encoder is further configured to encode the assigned attribute valueand the determined attribute correction values for first level of detailand the one or more additional levels of detail.

In some embodiments, the encoder is further configured to perform anupdate operation prior to encoding the assigned attribute correctionvalue and the determined attribute correction values for the first levelof detail and the one or more additional levels of detail. In someembodiments, an update operation may determine a relative importance ofa particular one of the determined attribute correction values on otherpoints in higher levels of detail of the hierarchical level of detailstructure. For example, in some situations slight errors in attributecorrection values for some points in one or more underlying levels ofdetail may have a greater impact on predicted attribute values forattributes of points in higher levels of detail. In such circumstances,more bits may be allocated (e.g. less quantization applied) todetermined attribute correction values for points with a greater impacton predicted attributes values of other points than are allocated todetermined attribute correction values for other points that have alesser impact on predicted attribute values for other points.Additionally, in some embodiments a transformation may be applied to theattribute values prior to performing an update operation and the effectsof the transformation operation may be taken into account whendetermining relative impacts of determined attribute correction valueson predicted attribute values for higher levels of detail.

In some embodiments, an encoder may further be configured to adaptivelychange a prediction strategy or number of nearest neighbors used in aprediction strategy based on variability of attributes for points in aneighborhood of a point being evaluated.

In some embodiments, a decoder is configured to receive compressedattribute information for a point cloud comprising at least one assignedattribute value for at least one point of the point cloud and dataindicating attribute correction values for attributes of the otherpoints of the point cloud. In some embodiments, the attribute correctionvalues may be ordered in a plurality of levels of detail for a pluralityof sub-sets of points of the point cloud. For example, the decoder mayreceive a compressed point cloud compressed by an encoder as describedabove. The decoder may further be configured to provide attributeinformation for a decompressed point cloud having a first level ofdetail and update the decompressed point cloud to include attributeinformation for additional sub-sets of points at one or more other onesof a plurality of levels of detail.

In some embodiments, wherein an update operation is performed at anencoder, a decoder may further “undo” the update operation uponreceiving compressed determined attribute correction values. Forexample, before applying a determined attribute correction value to apredicted attribute value, the decoder may account for the updateoperation performed on the determined attribute correction value.

In some embodiments, the decoder may be further configured to adaptivelychange a prediction strategy or number of nearest neighbors used in aprediction strategy based on variability of attributes for points in aneighborhood of a point being evaluated.

In some embodiments, a method comprises receiving compressed attributeinformation for a point cloud comprising at least one assigned attributevalue for at least one point of the point cloud and data indicatingattribute correction values for attributes of the other points of thepoint cloud. In some embodiments, the attribute correction values may beordered in a plurality of levels of detail for a plurality of sub-setsof the other points of the point cloud. The method may further includeproviding attribute information for a decompressed point cloud having afirst level of detail and updating the decompressed point cloud toinclude attribute information for additional sub-sets of points at oneor more other ones of a plurality of levels of detail. In someembodiments, the method may further include accounting for an updateoperation performed on attribute correction values for lower levels ofdetail before applying the attribute correction values to thedecompressed point cloud.

In some embodiments, one or more non-transitory computer-readable mediastore program instructions that, when executed by one or moreprocessors, cause the one or more processors to generate a hierarchicallevel of detail (LOD) structure as described herein to compressattribute information of a point cloud. In some embodiments, thenon-transitory computer-readable media further store, programinstructions that when executed by one or more processors, further causethe one or more processors to apply an update operation to attributecorrection values prior to encoding the attribute correction values,wherein the update operation is performed as described herein.

In some embodiments, one or more non-transitory computer-readable mediastore program instructions that, when executed by one or moreprocessors, cause the one or more processors to decode hierarchical LODstructure as described herein to decompress attribute information of apoint cloud. In some embodiments, the non-transitory computer-readablemedia further store, program instructions that when executed by one ormore processors, further cause the one or more processors to account foran update operation applied to attribute correction values at an encoderprior to applying decompressed attribute correction values to areconstructed point cloud, wherein accounting for the update operationis performed as described herein.

In some embodiments, a system includes a geometric encoder configured toperform quantization of points included in a point cloud, removeduplicate points, and perform octree encoding. To perform the octreeencoding, the geometry encoder is configured to use a binary arithmeticencoder to encode up to 256 occupancy symbols. The occupancy symbolsrepresent whether each of 8 sub-cubes of a cube are occupied with pointsor are un-occupied. Since there are 8 sub-cubes and two-possibilities(occupied or un-occupied), the number of occupancy symbols possible is2{circumflex over ( )}8 or 256. The geometric encoder may also beconfigured to utilize neighboring cubes or sub-cubes and a look-aheadprocedure using a look-ahead table to predict contexts for encoding theoccupancy symbols. For example, if neighboring cubes are empty, it ismore likely that the cube being encoded includes empty sub-cubes, andvice-versa.

Also, the possible number of encoding contexts may be reduced byassigning separate contexts to the most probable neighborhoodconfigurations and assigning less probable neighborhood configurationsto the same contexts, such that some of the less probable neighborhoodconfigurations share the same contexts. The system also include anattribute encoder configured to compress the attribute information forthe points. To compress the attribute information, the attribute encoderis configured to build a hierarchical level of detail (LOD) structure.For example, the attribute encoder may be configured to determine afirst level of detail for the attribute information of the point cloudand determine one or more additional levels of detail for the attributeinformation of the point cloud. To determine the first level of detailor the one or more additional levels of detail, the attribute encoder isconfigured to assign an attribute value to at least one point of thepoint cloud based on the attribute information included in the capturedpoint cloud for the point and, for a sub-set of respective ones of thepoints of the point cloud not included in a previously determined levelof detail: identify a set of neighboring points greater than a firstdistance from the point, determine a predicted attribute value for therespective point based on predicted or assigned attributes values forthe neighboring points, and determine, based on comparing the predictedattribute value for the respective point to the attribute informationfor the point included in the captured point cloud, an attributecorrection value for the point. The attribute encoder is furtherconfigured to encode the assigned attribute value and the determinedattribute correction values for first level of detail and the one ormore additional levels of detail. In some embodiments, the attributeencoder is further configured to adaptively change a prediction strategyor number of nearest neighbors used in a prediction strategy based onvariability of attributes for points in a neighborhood of a point beingevaluated.

In some embodiments, a decoder is configured to receive encodedgeometric information for a point cloud, apply an arithmetic decoder togenerate occupancy symbols (wherein the decoder decodes the geometryinformation encoded by the encoder as described above, also using abinary arithmetic decoder and look ahead tables). The occupancy symbolsare then used by an octree decoding element to recreate a point cloudgeometry of the point cloud being decompressed. The decoder is alsoconfigured to receive compressed attribute information for a point cloudcomprising at least one assigned attribute value for at least one pointof the point cloud and data indicating attribute correction values forattributes of the other points of the point cloud, wherein the attributecorrection values are ordered in a plurality of levels of detail for aplurality of sub-sets of the other points of the point cloud. Forexample, the decoder may receive a compressed point cloud compressed byan encoder as described above. The decoder may be further configured toprovide attribute information for a decompressed point cloud having afirst level of detail and update the decompressed point cloud to includeattribute information for additional sub-sets of points at one or moreother ones of the plurality of levels of detail.

In some embodiments, the decoder may be further configured to adaptivelychange a prediction strategy or number of nearest neighbors used in aprediction strategy based on variability of attributes for points in aneighborhood of a point being evaluated.

In some embodiments, a method comprises receiving encoded geometricinformation for a point cloud, applying an arithmetic decoder togenerate occupancy symbols (wherein the decoder decodes the geometryinformation encoded by the encoder as described above, also using abinary arithmetic decoder and look ahead tables). The occupancy symbolsare then used by an octree decoding element to recreate a point cloudgeometry of the point cloud being decompressed. In some embodiments, themethod further comprises receiving compressed attribute information fora point cloud comprising at least one assigned attribute value for atleast one point of the point cloud and data indicating attributecorrection values for attributes of the other points of the point cloud,wherein the attribute correction values are ordered in a plurality oflevels of detail for a plurality of sub-sets of the other points of thepoint cloud. The method further includes providing attribute informationfor a decompressed point cloud having a first level of detail andupdating the decompressed point cloud to include attribute informationfor additional sub-sets of points at one or more other ones of theplurality of levels of detail. In some embodiments, the method furtherincludes adaptively changing a prediction strategy or number of nearestneighbors used in a prediction strategy based on variability ofattributes for points in a neighborhood of a point being evaluated.

In some embodiments, one or more non-transitory computer-readable mediastore program instructions that, when executed by one or moreprocessors, cause the one or more processors to generate compressedgeometric information using an octree and binary arithmetic encoder withlook-ahead tables as described herein. In some embodiments, the computerreadable media store program instructions that, when executed by one ormore processors, cause the one or more processors to receivehierarchical level of detail (LOD) structure as described herein tocompress attribute information of a point cloud

In some embodiments, one or more non-transitory computer-readable mediastore program instructions that, when executed by one or moreprocessors, cause the one or more processors to decode geometric encodedinformation for a point cloud and to decompress a hierarchical LODstructure as described herein to provide attribute information for adecompressed point cloud.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a system comprising a sensor that capturesinformation for points of a point cloud and an encoder that compressesattribute information and/or spatial information of the point cloud,where the compressed point cloud information is sent to a decoder,according to some embodiments.

FIG. 1B illustrates a process for encoding attribute information of apoint cloud, according to some embodiments.

FIG. 1C illustrates representative views of point cloud information atdifferent stages of an encoding process, according to some embodiments.

FIG. 2A illustrates components of an encoder, according to someembodiments.

FIG. 2B illustrates components of a decoder, according to someembodiments.

FIG. 3 illustrates an example compressed attribute file, according tosome embodiments.

FIG. 4 illustrates a process for compressing attribute information of apoint cloud, according to some embodiments.

FIG. 5 illustrates a process for encoding attribute correction values,according to some embodiments.

FIGS. 6A-B illustrate an example process for compressing spatialinformation of a point cloud, according to some embodiments.

FIG. 7 illustrates another example process for compressing spatialinformation of a point cloud, according to some embodiments.

FIG. 8 illustrates an example process for decompressing compressedattribute information of a point cloud, according to some embodiments.

FIG. 9 illustrates components an example encoder that generates ahierarchical level of detail (LOD) structure, according to someembodiments.

FIG. 10 illustrates an example process for determining points to beincluded at different refinement layers of a level of detail (LOD)structure, according to some embodiments.

FIG. 11A illustrates an example level of detail (LOD) structure,according to some embodiments.

FIG. 11B illustrates an example compressed point cloud file comprisinglevel of details for a point cloud (LODs), according to someembodiments.

FIG. 12A illustrates a method of encoding attribute information of apoint cloud, according to some embodiments.

FIG. 12B illustrates a method of decoding attribute information of apoint cloud, according to some embodiments.

FIG. 12C illustrates example neighborhood configurations of cubes of anoctree, according to some embodiments.

FIG. 12D illustrates an example look-ahead cube, according to someembodiments.

FIG. 12E illustrates, an example of 31 contexts that may be used toadaptively encode an index value of a symbol S using a binary arithmeticencoder, according to some embodiments.

FIG. 12F illustrates an example octree compression technique using abinary arithmetic encoder, cache, and look-ahead table, according tosome embodiments.

FIG. 13A illustrates a direct transformation that may be applied at anencoder to encode attribute information of a point could, according tosome embodiments.

FIG. 13B illustrates an inverse transformation that may be applied at adecoder to decode attribute information of a point cloud, according tosome embodiments.

FIG. 14 illustrates compressed point cloud information being used in a3-D telepresence application, according to some embodiments.

FIG. 15 illustrates compressed point cloud information being used in avirtual reality application, according to some embodiments.

FIG. 16 illustrates an example computer system that may implement anencoder or decoder, according to some embodiments.

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure

“Comprising.” This term is open-ended. As used in the appended claims,this term does not foreclose additional structure or steps. Consider aclaim that recites: “An apparatus comprising one or more processor units. . . .” Such a claim does not foreclose the apparatus from includingadditional components (e.g., a network interface unit, graphicscircuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112(f), for that unit/circuit/component. Additionally,“configured to” can include generic structure (e.g., generic circuitry)that is manipulated by software and/or firmware (e.g., an FPGA or ageneral-purpose processor executing software) to operate in manner thatis capable of performing the task(s) at issue. “Configure to” may alsoinclude adapting a manufacturing process (e.g., a semiconductorfabrication facility) to fabricate devices (e.g., integrated circuits)that are adapted to implement or perform one or more tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On.” As used herein, this term is used to describe one or morefactors that affect a determination. This term does not forecloseadditional factors that may affect a determination. That is, adetermination may be solely based on those factors or based, at least inpart, on those factors. Consider the phrase “determine A based on B.”While in this case, B is a factor that affects the determination of A,such a phrase does not foreclose the determination of A from also beingbased on C. In other instances, A may be determined based solely on B.

DETAILED DESCRIPTION

As data acquisition and display technologies have become more advanced,the ability to capture point clouds comprising thousands or millions ofpoints in 2-D or 3-D space, such as via LIDAR systems, has increased.Also, the development of advanced display technologies, such as virtualreality or augmented reality systems, has increased potential uses forpoint clouds. However, point cloud files are often very large and may becostly and time-consuming to store and transmit. For example,communication of point clouds over private or public networks, such asthe Internet, may require considerable amounts of time and/or networkresources, such that some uses of point cloud data, such as real-timeuses, may be limited. Also, storage requirements of point cloud filesmay consume a significant amount of storage capacity of devices storingthe point cloud files, which may also limit potential applications forusing point cloud data.

In some embodiments, an encoder may be used to generate a compressedpoint cloud to reduce costs and time associated with storing andtransmitting large point cloud files. In some embodiments, a system mayinclude an encoder that compresses attribute information and/or spatialinformation (also referred to herein as geometry information) of a pointcloud file such that the point cloud file may be stored and transmittedmore quickly than non-compressed point clouds and in a manner such thatthe point cloud file may occupy less storage space than non-compressedpoint clouds. In some embodiments, compression of spatial informationand/or attributes of points in a point cloud may enable a point cloud tobe communicated over a network in real-time or in near real-time. Forexample, a system may include a sensor that captures spatial informationand/or attribute information about points in an environment where thesensor is located, wherein the captured points and correspondingattributes make up a point cloud. The system may also include an encoderthat compresses the captured point cloud attribute information. Thecompressed attribute information of the point cloud may be sent over anetwork in real-time or near real-time to a decoder that decompressesthe compressed attribute information of the point cloud. Thedecompressed point cloud may be further processed, for example to make acontrol decision based on the surrounding environment at the location ofthe sensor. The control decision may then be communicated back to adevice at or near the location of the sensor, wherein the devicereceiving the control decision implements the control decision inreal-time or near real-time. In some embodiments, the decoder may beassociated with an augmented reality system and the decompressedattribute information may be displayed or otherwise used by theaugmented reality system. In some embodiments, compressed attributeinformation for a point cloud may be sent with compressed spatialinformation for points of the point cloud. In other embodiments, spatialinformation and attribute information may be separately encoded and/orseparately transmitted to a decoder.

In some embodiments, a system may include a decoder that receives one ormore point cloud files comprising compressed attribute information via anetwork from a remote server or other storage device that stores the oneor more point cloud files. For example, a 3-D display, a holographicdisplay, or a head-mounted display may be manipulated in real-time ornear real-time to show different portions of a virtual world representedby point clouds. In order to update the 3-D display, the holographicdisplay, or the head-mounted display, a system associated with thedecoder may request point cloud files from the remote server based onuser manipulations of the displays, and the point cloud files may betransmitted from the remote server to the decoder and decoded by thedecoder in real-time or near real-time. The displays may then be updatedwith updated point cloud data responsive to the user manipulations, suchas updated point attributes.

In some embodiments, a system, may include one or more LIDAR systems,3-D cameras, 3-D scanners, etc., and such sensor devices may capturespatial information, such as X, Y, and Z coordinates for points in aview of the sensor devices. In some embodiments, the spatial informationmay be relative to a local coordinate system or may be relative to aglobal coordinate system (for example, a Cartesian coordinate system mayhave a fixed reference point, such as a fixed point on the earth, or mayhave a non-fixed local reference point, such as a sensor location).

In some embodiments, such sensors may also capture attribute informationfor one or more points, such as color attributes, reflectivityattributes, velocity attributes, acceleration attributes, timeattributes, modalities, and/or various other attributes. In someembodiments, other sensors, in addition to LIDAR systems, 3-D cameras,3-D scanners, etc., may capture attribute information to be included ina point cloud. For example, in some embodiments, a gyroscope oraccelerometer, may capture motion information to be included in a pointcloud as an attribute associated with one or more points of the pointcloud. For example, a vehicle equipped with a LIDAR system, a 3-Dcamera, or a 3-D scanner may include the vehicle's direction and speedin a point cloud captured by the LIDAR system, the 3-D camera, or the3-D scanner. For example, when points in a view of the vehicle arecaptured they may be included in a point cloud, wherein the point cloudincludes the captured points and associated motion informationcorresponding to a state of the vehicle when the points were captured.

In some embodiments, attribute information may comprise string values,such as different modalities. For example attribute information mayinclude string values indicating a modality such as “walking”,“running”, “driving”, etc. In some embodiments, an encoder may comprisea “string-value” to integer index, wherein certain strings areassociated with certain corresponding integer values. In someembodiments, a point cloud may indicate a string value for a point byincluding an integer associated with the string value as an attribute ofthe point. The encoder and decoder may both store a common string valueto integer index, such that the decoder can determine string values forpoints based on looking up the integer value of the string attribute ofthe point in a string value to integer index of the decoder that matchesor is similar to the string value to integer index of the encoder.

In some embodiments, an encoder compresses and encodes spatialinformation of a point cloud to compress the spatial information inaddition to compressing attribute information for attributes of thepoints of the point cloud. For example, to compress spatial informationa K-D tree may be generated wherein, respective numbers of pointsincluded in each of the cells of the K-D tree are encoded. This sequenceof encoded point counts may encode spatial information for points of apoint cloud. Also, in some embodiments, a sub-sampling and predictionmethod may be used to compress and encode spatial information for apoint cloud. In some embodiments, the spatial information may bequantized prior to being compressed and encoded. Also, in someembodiments, compression of spatial information may be lossless. Thus, adecoder may be able to determine a same view of the spatial informationas an encoder. Also, an encoder may be able to determine a view of thespatial information a decoder will encounter once the compressed spatialinformation is decoded. Because, both an encoder and decoder may have orbe able to recreate the same spatial information for the point cloud,spatial relationships may be used to compress attribute information forthe point cloud.

For example, in many point clouds, attribute information betweenadjacent points or points that are located at relatively short distancesfrom each other may have high levels of correlation between attributes,and thus relatively small differences in point attribute values. Forexample, proximate points in a point cloud may have relatively smalldifferences in color, when considered relative to points in the pointcloud that are further apart.

In some embodiments, an encoder may include a predictor that determinesa predicted attribute value of an attribute of a point in a point cloudbased on attribute values for similar attributes of neighboring pointsin the point cloud and based on respective distances between the pointbeing evaluated and the neighboring points. In some embodiments,attribute values of attributes of neighboring points that are closer toa point being evaluated may be given a higher weighting than attributevalues of attributes of neighboring points that are further away fromthe point being evaluated. Also, the encoder may compare a predictedattribute value to an actual attribute value for an attribute of thepoint in the original point cloud prior to compression. A residualdifference, also referred to herein as an “attribute correction value”may be determined based on this comparison. An attribute correctionvalue may be encoded and included in compressed attribute informationfor the point cloud, wherein a decoder uses the encoded attributecorrection value to correct a predicted attribute value for the point,wherein the attribute value is predicted using a same or similarprediction methodology at the decoder that is the same or similar to theprediction methodology that was used at the encoder.

In some embodiments, to encode attribute values an encoder may generatea minimum spanning tree for points of a point cloud based on spatialinformation for the points of the point cloud. The encoder may select afirst point as a starting point and may determine an evaluation orderfor other ones of the points of the point cloud based on minimumdistances from the starting point to a closest neighboring point, and asubsequent minimum distance from the neighboring point to the nextclosest neighboring point, etc. In this way, an evaluation order fordetermining predicted attribute values of the points of the point cloudmay be determined. Because the decoder may receive or re-create the samespatial information as the spatial information used by the encoder, thedecoder may generate the same minimum spanning tree for the point cloudand may determine the same evaluation order for the points of the pointcloud.

In some embodiments, an encoder may assign an attribute value for astarting point of a point cloud to be used to generate a minimumspanning tree. An encoder may predict an attribute value for a nextnearest point to the starting point based on the attribute value of thestarting point and a distance between the starting point and the nextnearest point. The encoder may then determine a difference between thepredicted attribute value for the next nearest point and the actualattribute value for the next nearest point included in thenon-compressed original point cloud. This difference may be encoded in acompressed attribute information file as an attribute correction valuefor the next nearest point. The encoder may then repeat a similarprocess for each point in the evaluation order. To predict the attributevalue for subsequent points in the evaluation order, the encoder mayidentify the K-nearest neighboring points to a particular point beingevaluated, wherein the identified K-nearest neighboring points haveassigned or predicted attribute values. In some embodiments, “K” may bea configurable parameter that is communicated from an encoder to adecoder.

The encoder may determine a distance in X, Y, and Z space between apoint being evaluated and each of the identified neighboring points. Forexample, the encoder may determine respective Euclidian distances fromthe point being evaluated to each of the neighboring points. The encodermay then predict an attribute value for an attribute of the point beingevaluated based on the attribute values of the neighboring points,wherein the attribute values of the neighboring points are weightedaccording to an inverse of the distances from the point being evaluatedto the respective ones of the neighboring points. For example, attributevalues of neighboring points that are closer to the point beingevaluated may be given more weight than attribute values of neighboringpoints that are further away from the point being evaluated.

In a similar manner as described for the first neighboring point, theencoder may compare a predicted value for each of the other points ofthe point cloud to an actual attribute value in an originalnon-compressed point cloud, for example the captured point cloud. Thedifference may be encoded as an attribute correction value for anattribute of one of the other points that is being evaluated. In someembodiments, attribute correction values may be encoded in an order in acompressed attribute information file in accordance with the evaluationorder determined based on the minimum spanning tree. Because the encoderand the decoder may determine the same evaluation order based on thespatial information for the point cloud, the decoder may determine whichattribute correction value corresponds to which attribute of which pointbased on the order in which the attribute correction values are encodedin the compressed attribute information file. Additionally, the startingpoint and one or more attribute value(s) of the starting point may beexplicitly encoded in a compressed attribute information file such thatthe decoder may determine the evaluation order starting with the samepoint as was used to start the evaluation order at the encoder.Additionally, the one or more attribute value(s) of the starting pointmay provide a value of a neighboring point that a decoder uses todetermine a predicted attribute value for a point being evaluated thatis a neighboring point to the starting point

In some embodiments, an encoder may determine a predicted value for anattribute of a point based on temporal considerations. For example, inaddition to or in place of determining a predicted value based onneighboring points in a same “frame” e.g. point in time as the pointbeing evaluated, the encoder may consider attribute values of the pointin adjacent and subsequent time frames.

FIG. 1A illustrates a system comprising a sensor that capturesinformation for points of a point cloud and an encoder that compressesattribute information of the point cloud, where the compressed attributeinformation is sent to a decoder, according to some embodiments.

System 100 includes sensor 102 and encoder 104. Sensor 102 captures apoint cloud 110 comprising points representing structure 106 in view 108of sensor 102. For example, in some embodiments, structure 106 may be amountain range, a building, a sign, an environment surrounding a street,or any other type of structure. In some embodiments, a captured pointcloud, such as captured point cloud 110, may include spatial andattribute information for the points included in the point cloud. Forexample, point A of captured point cloud 110 comprises X, Y, Zcoordinates and attributes 1, 2, and 3. In some embodiments, attributesof a point may include attributes such as R, G, B color values, avelocity at the point, an acceleration at the point, a reflectance ofthe structure at the point, a time stamp indicating when the point wascaptured, a string-value indicating a modality when the point wascaptured, for example “walking”, or other attributes. The captured pointcloud 110 may be provided to encoder 104, wherein encoder 104 generatesa compressed version of the point cloud (compressed attributeinformation 112) that is transmitted via network 114 to decoder 116. Insome embodiments, a compressed version of the point cloud, such ascompressed attribute information 112, may be included in a commoncompressed point cloud that also includes compressed spatial informationfor the points of the point cloud or, in some embodiments, compressedspatial information and compressed attribute information may becommunicated as separate files.

In some embodiments, encoder 104 may be integrated with sensor 102. Forexample, encoder 104 may be implemented in hardware or software includedin a sensor device, such as sensor 102. In other embodiments, encoder104 may be implemented on a separate computing device that is proximateto sensor 102.

FIG. 1B illustrates a process for encoding compressed attributeinformation of a point cloud, according to some embodiments. Also, FIG.1C illustrates representative views of point cloud information atdifferent stages of an encoding process, according to some embodiments.

At 152, an encoder, such as encoder 104, receives a captured point cloudor a generated point cloud. For example, in some embodiments a pointcloud may be captured via one or more sensors, such as sensor 102, ormay be generated in software, such as in a virtual reality or augmentedreality system. For example, 164 illustrates an example captured orgenerated point cloud. Each point in the point cloud shown in 164 mayhave one or more attributes associated with the point. Note that pointcloud 164 is shown in 2D for ease of illustration, but may includepoints in 3D space.

At 154, a minimum spanning tree is determined based on the spatialinformation of the point cloud received by the encoder at 152. In orderto determine a minimum spanning tree, a minimum spanning tree generatorof an encoder may select a starting point for the minimum spanning tree.The minimum spanning tree generator may then identify points that areadjacent to the starting point. The adjacent points may then be sortedbased on respective distances between the respective identified adjacentpoints and the starting point. The adjacent point that is at theshortest distance from the starting point, may be selected as the nextpoint to be visited. A “weight” of an “edge”, e.g. a distance betweenpoints in a point cloud, may be determined for an edge between thestarting point and the adjacent point selected to be next visited,wherein, longer distances are given greater weights than shorterdistances. After the adjacent point closest to the starting point isadded to the minimum spanning tree, the adjacent point may then beevaluated and points adjacent to the point currently being evaluated(e.g. the point that was previously selected to be next visited) may beidentified. The identified adjacent points may be sorted based onrespective distances between the point currently being evaluated and theidentified adjacent points. The adjacent point at the shortest distance,e.g. “edge”, from the point currently being evaluated may be selected asthe next point to be included in the minimum spanning tree. A weight forthe edge between the point currently being evaluated and the nextselected adjacent point may be determined and added to the minimumspanning tree. A similar process may be repeated for each of the otherpoints of the point cloud to generate a minimum spanning tree for thepoint cloud

For example, 166 illustrates an illustration of a minimum spanning tree.In the minimum spanning tree shown in 166, each vertex may represent apoint in a point cloud, and the edge weights between vertices, forexample, 1, 2, 3, 4, 7, 8, etc. may represent distances between pointsin the point cloud. For example a distance between vertex 172 and vertex174 may have a weight of 7, whereas a distance between vertices 172 and176 may have a weight of 8. This may indicate that a distance in a pointcloud between a point corresponding to vertex 172 and a pointcorresponding to vertex 176 is greater than a distance in the pointcloud between a point corresponding to vertex 172 and a pointcorresponding to vertex 174. In some embodiments, weights shown in aminimum spanning tree may be based on vector distances in 3-D space,such as Euclidean distances.

At 156, an attribute value for one or more attributes of a startingpoint, such as the starting point used to generate the minimum spanningtree, may be assigned to be encoded and included in compressed attributeinformation for the point cloud. As discussed above, predicted attributevalues for points of a point cloud may be determined based on attributevalues of neighboring points. However, an initial attribute value for atleast one point is provided to a decoder so that the decoder maydetermine attribute values for other points using at least the initialattribute value and attribute correction values for correcting predictedattribute values that are predicted based on the initial attributevalue. Thus, one or more attribute values for at least one startingpoint are explicitly encoded in a compressed attribute information file.Additionally, spatial information for the starting point may beexplicitly encoded such that a minimum spanning tree generator of adecoder may determine which point of the points of the point cloud is tobe used as a starting point for generating a minimum spanning tree. Insome embodiments, a starting point may be indicated in other ways otherthan explicitly encoding the spatial information for the starting point,such as flagging the starting point or other methods of pointidentification.

Because a decoder will receive an indication of a starting point andwill encounter the same or similar spatial information for the points ofthe point cloud as the encoder, the decoder may determine a same minimumspanning tree from the same starting point as was determined by theencoder. Additionally, the decoder may determine a same processing orderas the encoder based on the same minimum spanning tree determined by thedecoder.

At 158, for a current point being evaluated, a prediction/correctionevaluator of an encoder determines a predicted attribute value for anattribute of the point currently being evaluated. In some embodiments, apoint currently being evaluated may have more than one attribute.Accordingly, a prediction/correction evaluator of an encoder may predictmore than one attribute value for the point. For each point beingevaluated, the prediction/correction evaluator may identify a set ofnearest neighboring points that have assigned or predicted attributevalues. In some embodiments, a number of neighboring points to identify,“K”, may be a configurable parameter of an encoder and the encoder mayinclude configuration information in a compressed attribute informationfile indicating the parameter “K” such that a decoder may identify asame number of neighboring points when performing attribute prediction.The prediction/correction evaluator may then use weights from theminimum spanning tree or may otherwise determine distances between thepoint being evaluated and respective ones of the identified neighboringpoints. The prediction/correction evaluator may use an inverse distanceinterpolation method to predict an attribute value for each attribute ofthe point being evaluated. The prediction/correction evaluator may thenpredict an attribute value of the point being evaluated based on anaverage of inverse-distance weighted attribute values of the identifiedneighboring points.

For example, 168 illustrates a point (X,Y,Z) being evaluated whereinattribute A1 is being determined based on inverse distance weightedattribute values of eight identified neighboring points.

At 160, an attribute correction value is determined for each point. Theattribute correction value is determined based on comparing a predictedattribute value for each attribute of a point to corresponding attributevalues of the point in an original non-compressed point cloud, such asthe captured point cloud. For example, 170 illustrates an equation fordetermining attribute correction values, wherein a captured value issubtracted from a predicted value to determine an attribute correctionvalue. Note that while, FIG. 1B shows attribute values being predictedat 158 and attribute correction values being determined at 160, in someembodiments attribute correction values may be determined for a pointsubsequent to predicting an attribute value for the point. A next pointmay then be evaluated, wherein a predicted attribute value is determinedfor the point and an attribute correction value is determined for thepoint. Thus 158 and 160 may be repeated for each point being evaluated.In other embodiments, predicted values may be determined for multiplepoints and then attribute correction values may be determined. In someembodiments, predictions for subsequent points being evaluated may bebased on predicted attribute values or may be based on correctedattribute values or both. In some embodiments, both an encoder and adecoder may follow the same rules as to whether predicted values forsubsequent points are to be determined based on predicted or correctedattribute values.

At 162, the determined attribute correction values for the points of thepoint cloud, one or more assigned attribute values for the startingpoint, spatial information or other indicia of the starting point, andany configuration information to be included in a compressed attributeinformation file is encoded. As discussed in more detail in FIG. 5various encoding methods, such as arithmetic encoding and/or Golombencoding may be used to encode the attribute correction values, assignedattribute values, and the configuration information.

FIG. 2A illustrates components of an encoder, according to someembodiments.

Encoder 202 may be a similar encoder as encoder 104 illustrated in FIG.1A. Encoder 202 includes spatial encoder 204, minimum spanning treegenerator 210, prediction/correction evaluator 206, incoming datainterface 214, and outgoing data interface 208. Encoder 202 alsoincludes context store 216 and configuration store 218.

In some embodiments, a spatial encoder, such as spatial encoder 204, maycompress spatial information associated with points of a point cloud,such that the spatial information can be stored or transmitted in acompressed format. In some embodiments, a spatial encoder, may utilizeK-D trees to compress spatial information for points of a point cloud asdiscussed in more detail in regard to FIG. 7. Also, in some embodiments,a spatial encoder, such as spatial encoder 204, may utilize asub-sampling and prediction technique as discussed in more detail inregard to FIGS. 6A-B. In some embodiments, a spatial encoder, such asspatial encoder 204, may utilize Octrees to compress spatial informationfor points of a point cloud as discussed in more detail in regard toFIG. 11D.

In some embodiments, compressed spatial information may be stored ortransmitted with compressed attribute information or may be stored ortransmitted separately. In either case, a decoder receiving compressedattribute information for points of a point cloud may also receivecompressed spatial information for the points of the point cloud, or mayotherwise obtain the spatial information for the points of the pointcloud.

A minimum spanning tree generator, such as minimum spanning treegenerator 210, may utilize spatial information for points of a pointcloud to generate a minimum spanning tree representing minimum distancesbetween points of the point cloud. Because a decoder is provided orotherwise obtains the same spatial information for points of a pointcloud as are available at the encoder, a minimum spanning treedetermined by a minimum spanning tree generator of an encoder, such asminimum spanning tree generator 210 of encoder 202, may be the same orsimilar as a minimum spanning tree generated by a minimum spanning treegenerator of a decoder, such as minimum spanning tree generator 228 ofdecoder 220.

A prediction/correction evaluator, such as prediction/correctionevaluator 206 of encoder 202, may determine predicted attribute valuesfor points of a point cloud based on an inverse distance interpolationmethod using attribute values of the K-nearest neighboring points of apoint for whom an attribute value is being predicted. Theprediction/correction evaluator may also compare a predicted attributevalue of a point being evaluated to an original attribute value of thepoint in a non-compressed point cloud to determine an attributecorrection value.

An outgoing data encoder, such as outgoing data encoder 208 of encoder202, may encode attribute correction values and assigned attributevalues included in a compressed attribute information file for a pointcloud. In some embodiments, an outgoing data encoder, such as outgoingdata encoder 208, may select an encoding context for encoding a value,such as an assigned attribute value or an attribute correction value,based on a number of symbols included in the value. In some embodiments,values with more symbols may be encoded using an encoding contextcomprising Golomb exponential encoding, whereas values with fewersymbols may be encoded using arithmetic encoding. In some embodiments,encoding contexts may include more than one encoding technique. Forexample, a portion of a value may be encoded using arithmetic encodingwhile another portion of the value may be encoded using Golombexponential encoding. In some embodiments, an encoder, such as encoder202, may include a context store, such as context store 216, that storesencoding contexts used by an outgoing data encoder, such as outgoingdata encoder 208, to encode attribute correction values and assignedattribute values.

In some embodiments, an encoder, such as encoder 202, may also includean incoming data interface, such as incoming data interface 214. In someembodiments, an encoder may receive incoming data from one or moresensors that capture points of a point cloud or that capture attributeinformation to be associated with points of a point cloud. For example,in some embodiments, an encoder may receive data from an LIDAR system,3-D-camera, 3-D scanner, etc. and may also receive data from othersensors, such as a gyroscope, accelerometer, etc. Additionally, anencoder may receive other data such as a current time from a systemclock, etc. In some embodiments, such different types of data may bereceived by an encoder via an incoming data interface, such as incomingdata interface 214 of encoder 202.

In some embodiments, an encoder, such as encoder 202, may furtherinclude a configuration interface, such as configuration interface 212,wherein one or more parameters used by the encoder to compress a pointcloud may be adjusted via the configuration interface. In someembodiments, a configuration interface, such as configuration interface212, may be a programmatic interface, such as an API. Configurationsused by an encoder, such as encoder 202, may be stored in aconfiguration store, such as configuration store 218.

In some embodiments, an encoder, such as encoder 202, may include moreor fewer components than shown in FIG. 2A.

FIG. 2B illustrates components of a decoder, according to someembodiments.

Decoder 220 may be a similar decoder as decoder 116 illustrated in FIG.1A. Decoder 220 includes encoded data interface 226, spatial decoder222, minimum spanning tree generator 228, prediction evaluator 224,context store 232, configuration store 234, and decoded data interface220.

A decoder, such as decoder 220, may receive an encoded compressed pointcloud and/or an encoded compressed attribute information file for pointsof a point cloud. For example, a decoder, such as decoder 220, mayreceive a compressed attribute information file, such a compressedattribute information 112 illustrated in FIG. 1A or compressed attributeinformation file 300 illustrated in FIG. 3. The compressed attributeinformation file may be received by a decoder via an encoded datainterface, such as encoded data interface 226. The encoded compressedpoint cloud may be used by the decoder to determine spatial informationfor points of the point cloud. For example, spatial information ofpoints of a point cloud included in a compressed point cloud may begenerated by a spatial information generator, such as spatialinformation generator 222. In some embodiments, a compressed point cloudmay be received via an encoded data interface, such as encoded datainterface 226, from a storage device or other intermediary source,wherein the compressed point cloud was previously encoded by an encoder,such as encoder 104. In some embodiments, an encoded data interface,such as encoded data interface 226, may decode spatial information. Forexample the spatial information may have been encoded using variousencoding techniques such as arithmetic encoding, Golomb encoding, etc. Aspatial information generator, such as spatial information generator222, may receive decoded spatial information from an encoded datainterface, such as encoded data interface 226, and may use the decodedspatial information to generate a representation of the geometry of thepoint cloud being de-compressed. For example, decoded spatialinformation may be formatted as residual values to be used in asub-sampled prediction method to recreate a geometry of a point cloud tobe decompressed. In such situations, the spatial information generator222, may recreate the geometry of the point cloud being decompressedusing decoded spatial information from encoded data interface 226, andminimum spanning tree generator 228 may determine a minimum spanningtree for the point cloud being decompressed based on the recreatedgeometry for the point cloud being decompressed generated by spatialinformation generator 222.

Once spatial information for a point cloud is determined, a minimumspanning tree generator, such as minimum spanning tree generator 228,may generate a minimum spanning tree based on the spatial informationfor the point cloud. The minimum spanning tree may be used by aprediction evaluator of a decoder, such as prediction evaluator 224 ofdecoder 220, to determine an evaluation order for determining attributevalues of points of the point cloud. Additionally, the minimum spanningtree may be used by a prediction evaluator, such as prediction evaluator224, to identify nearest neighboring points to a point being evaluated.

A prediction evaluator of a decoder, such as prediction evaluator 224,may select a starting point of a minimum spanning tree based on anassigned starting point included in a compressed attribute informationfile. In some embodiments, the compressed attribute information file mayinclude one or more assigned values for one or more correspondingattributes of the starting point. In some embodiments, a predictionevaluator, such as prediction evaluator 224, may assign values to one ormore attributes of a starting point in a decompressed model of a pointcloud being decompressed based on assigned values for the starting pointincluded in a compressed attribute information file. A predictionevaluator, such as prediction evaluator 224, may further utilize theassigned values of the attributes of the starting point to determineattribute values of neighboring points. For example, a predictionevaluator may select a next nearest neighboring point to the startingpoint as a next point to evaluate, wherein the next nearest neighboringpoint is selected based on a shortest distance to a neighboring pointfrom the starting point in the minimum spanning tree. Note that becausethe minimum spanning tree is generated based on the same or similarspatial information at the decoder as was used to generate a minimumspanning tree at an encoder, the decoder may determine the sameevaluation order for evaluating the points of the point cloud beingdecompressed as was determined at the encoder by identifying nextnearest neighbors in the minimum spanning tree.

Once the prediction evaluator has identified the “K” nearest neighboringpoints to a point being evaluated, the prediction evaluator may predictone or more attribute values for one or more attributes of the pointbeing evaluated based on attribute values of corresponding attributes ofthe “K” nearest neighboring points. In some embodiments, an inversedistance interpolation technique may be used to predict an attributevalue of a point being evaluated based on attribute values ofneighboring points, wherein attribute values of neighboring points thatare at a closer distance to the point being evaluated are weighted moreheavily than attribute values of neighboring points that are at furtherdistances from the point being evaluated.

A prediction evaluator, such as prediction evaluator 224, may apply anattribute correction value to a predicted attribute value to determinean attribute value to include for the point in a decompressed pointcloud. In some embodiments, an attribute correction value for anattribute of a point may be included in a compressed attributeinformation file. In some embodiments, attribute correction values maybe encoded using one of a plurality of supported coding contexts,wherein different coding contexts are selected to encode differentattribute correction values based on a number of symbols included in theattribute correction value. In some embodiments, a decoder, such asdecoder 220, may include a context store, such as context store 232,wherein the context store stores a plurality of encoding context thatmay be used to decode assigned attribute values or attribute correctionvalues that have been encoded using corresponding encoding contexts atan encoder.

A decoder, such as decoder 220, may provide a decompressed point cloudgenerated based on a received compressed point cloud and/or a receivedcompressed attribute information file to a receiving device orapplication via a decoded data interface, such as decoded data interface230. The decompressed point cloud may include the points of the pointcloud and attribute values for attributes of the points of the pointcloud. In some embodiments, a decoder may decode some attribute valuesfor attributes of a point cloud without decoding other attribute valuesfor other attributes of a point cloud. For example, a point cloud mayinclude color attributes for points of the point cloud and may alsoinclude other attributes for the points of the point cloud, such asvelocity, for example. In such a situation, a decoder may decode one ormore attributes of the points of the point cloud, such as the velocityattribute, without decoding other attributes of the points of the pointcloud, such as the color attributes.

In some embodiments, the decompressed point cloud and/or decompressedattribute information file may be used to generate a visual display,such as for a head mounted display. Also, in some embodiments, thedecompressed point cloud and/or decompressed attribute information filemay be provided to a decision making engine that uses the decompressedpoint cloud and/or decompressed attribute information file to make oneor more control decisions. In some embodiments, the decompressed pointcloud and/or decompressed attribute information file may be used invarious other applications or for various other purposes.

FIG. 3 illustrates an example compressed attribute information file,according to some embodiments. Attribute information file 300 includesconfiguration information 302, point cloud data 304, and point attributecorrection values 306. In some embodiments, point cloud file 300 may becommunicated in parts via multiple packets. In some embodiments, not allof the sections shown in attribute information file 300 may be includedin each packet transmitting compressed attribute information. In someembodiments, an attribute information file, such as attributeinformation file 300, may be stored in a storage device, such as aserver that implements an encoder or decoder, or other computing device.

FIG. 4 illustrates a process for compressing attribute information of apoint cloud, according to some embodiments.

At 402, an encoder receives a point cloud that includes attributeinformation for at least some of the points of the point cloud. Thepoint cloud may be received from one or more sensors that capture thepoint cloud, or the point cloud may be generated in software. Forexample, a virtual reality or augmented reality system may havegenerated the point cloud

At 404, the spatial information of the point cloud, for example X, Y,and Z coordinates for the points of the point cloud may be quantized. Insome embodiments, coordinates may be rounded off to the nearestmeasurement unit, such as a meter, centimeter, millimeter, etc.

At 406, the quantized spatial information is compressed. In someembodiments, spatial information may be compressed using a sub-samplingand subdivision prediction technique as discussed in more detail inregard to FIGS. 6A-B. Also, in some embodiments, spatial information maybe compressed using a K-D tree compression technique as discussed inmore detail in regard to FIG. 7, or may be compressed using an Octreecompression technique as discussed in more detail in regard to FIGS.11C-D. In some embodiments, other suitable compression techniques may beused to compress spatial information of a point cloud.

At 408, the compressed spatial information for the point cloud isencoded as a compressed point cloud file or a portion of a compressedpoint cloud file. In some embodiments, compressed spatial informationand compressed attribute information may be included in a commoncompressed point cloud file, or may be communicated or stored asseparate files.

At 412, the received spatial information of the point cloud is used togenerate a minimum spanning tree. In some embodiments, the spatialinformation of the point cloud may be quantized before generating theminimum spanning tree. Additionally, in some embodiments wherein a lossycompression technique is used to compress the spatial information of thepoint cloud, the spatial information may be lossy encoded and lossydecoded prior to generating the minimum spanning tree. In embodimentsthat utilize lossy compression for spatial information, encoding anddecoding the spatial information at the encoder may ensure that aminimum spanning tree generated at the encoder will match a minimumspanning tree that will be generated at a decoder using decoded spatialinformation that was previously lossy encoded.

Additionally, in some embodiments, at 410, attribute information forpoints of the point cloud may be quantized. For example attribute valuesmay be rounded to whole numbers or to particular measurement increments.In some embodiments wherein attribute values are integers, such as whenintegers are used to communicate string values, such as “walking”,“running”, “driving”, etc., quantization at 410 may be omitted.

At 414, attribute values for a starting point are assigned. The assignedattribute values for the starting point are encoded in a compressedattribute information file along with attribute correction values.Because a decoder predicts attribute values based on distances toneighboring points and attribute values of neighboring points, at leastone attribute value for at least one point is explicitly encoded in acompressed attribute file. In some embodiments, points of a point cloudmay comprise multiple attributes and at least one attribute value foreach type of attribute may be encoded for at least one point of thepoint cloud, in such embodiments. In some embodiments, a starting pointmay be a first point evaluated when determining the minimum spanningtree at 412. In some embodiments, an encoder may encode data indicatingspatial information for a starting point and/or other indicia of whichpoint of the point cloud is the starting point or starting points.Additionally, the encoder may encode attribute values for one or moreattributes of the starting point.

At 416, the encoder determines an evaluation order for predictingattribute values for other points of the point cloud, other than thestarting point, said predicting and determining attribute correctionvalues, may be referred to herein as “evaluating” attributes of a point.The evaluation order may be determined based on a shortest distance fromthe starting point to an adjacent neighboring point, wherein the closestneighboring point is selected as the next point in the evaluation order.In some embodiments, an evaluation order may be determined only for anext point to evaluate. In other embodiments, an evaluation order forall or multiple ones of the points of the point cloud may be determinedat 416. In some embodiments, an evaluation order may be determined onthe fly, e.g. one point at a time as the points are evaluated.

At 418, a neighboring point of the starting point or of a subsequentpoint being evaluated is selected. In some embodiments, a neighboringpoint to be next evaluated may be selected based on the neighboringpoint being at a shortest distance from a point last evaluated, ascompared to other neighboring points of the point last evaluated. Insome embodiments, a point selected at 418 may be selected based on anevaluation order determined at 416. In some embodiments, an evaluationorder may be determined on the fly, e.g. one point at a time as thepoints are evaluated. For example, a next point in the evaluation ordermay be determined each time a next point to be evaluated is selected at418. In such embodiments, 416 may be omitted. Because points areevaluated in an order wherein each next point to be evaluated is at ashortest distance from a point last evaluated, entropy between attributevalues of the points being evaluated may be minimized. This is becausepoints adjacent to one another are most likely to have similarattributes. Though in some circumstances, adjacent points may havevarying levels of similarity between attributes.

At 420, the “K” nearest neighboring points to the point currently beingevaluated are determined. The parameter “K” may be a configurableparameter selected by an encoder or provided to an encoder as a userconfigurable parameter. In order to select the “K” nearest neighboringpoints, an encoder may identify the first “K” nearest points to a pointbeing evaluated according to the minimum spanning tree determined at412. In some embodiments, only points having assigned attribute valuesor for which predicted attribute values have already been determined maybe included in the “K” nearest neighboring points. In some embodimentsvarious numbers of points may identified. For example, in someembodiments, “K” may be 5 points, 10 points, 16 points, etc. Because apoint cloud comprises points in 3-D space a particular point may havemultiple neighboring points in multiple planes. In some embodiments, anencoder and a decoder may be configured to identify points as the “K”nearest neighboring points regardless of whether or not a value hasalready been predicted for the point. Also, in some embodiments,attribute values for points used in predication may be previouslypredicted attribute values or corrected predicted attribute values thathave been corrected based on applying an attribute correction value. Ineither case, an encoder and a decoder may be configured to apply thesame rules when identifying the “K” nearest neighboring points and whenpredicting an attribute value of a point based on attribute values ofthe “K” nearest neighboring points.

At 422, one or more attribute values are determined for each attributeof the point currently being evaluated. The attribute values may bedetermined based on an inverse distance interpolation. The inversedistance interpolation may interpolate the predicted attribute valuebased on the attribute values of the “K” nearest neighboring points. Theattribute values of the “K” nearest neighboring points may be weightedbased on respective distances between respective ones of the “K” nearestneighboring points and the point being evaluated. Attribute values ofneighboring points that are at shorter distances from the pointcurrently being evaluated may be weighted more heavily than attributevalues of neighboring points that are at greater distances from thepoint currently being evaluated.

At 424, attribute correction values are determined for the one or morepredicted attribute values for the point currently being evaluated. Theattribute correction values may be determined based on comparing thepredicted attribute values to corresponding attribute values for thesame point (or a similar point) in the point cloud prior to attributeinformation compression. In some embodiments, quantized attributeinformation, such as the quantized attribute information generated at410, may be used to determine attribute correction values. In someembodiments, an attribute correction value may also be referred to as a“residual error” wherein the residual error indicates a differencebetween a predicted attribute value and an actual attribute value.

At 426, it is determined if there are additional points in the pointcloud for which attribute correction values are to be determined. Ifthere are additional points to evaluate, the process reverts to 418 andthe next point in the evaluation order is selected to be evaluated. Asdiscussed above, in some embodiments an evaluation order may bedetermined on the fly, e.g. one point at a time as the points areevaluated. Thus, in such embodiments, a minimum spanning tree may beconsulted to select a next point to evaluate based on the next pointbeing at the shortest distance from the point last evaluated. Theprocess may repeat steps 418-426 until all or a portion of all of thepoints of the point cloud have been evaluated to determine predictedattribute values and attribute correction values for the predictedattribute values.

At 428, the determined attribute correction values, the assignedattribute values, and any configuration information for decoding thecompressed attribute information file, such as a parameter “K”, isencoded.

The attribute correction values, the assigned attribute values, and anyconfiguration information may be encoded using various encodingtechniques.

For example, FIG. 5 illustrates a process for encoding attributecorrection values, according to some embodiments. At 502, an attributecorrection value for a point whose values (e.g. attribute correctionvalues) are being encoded is converted to an unsigned value. Forexample, in some embodiments, attribute correction values that arenegative values may be assigned odd numbers and attribute correctionvalues that are positive values may be assigned even numbers. Thus,whether or not the attribute correction value is positive or negativemay be implied based on whether or not a value of the attributecorrection value is an even number or an odd number. In someembodiments, assigned attribute values may also be converted intounsigned values. In some embodiments, attribute values may all bepositive values, for example in the case of integers that are assignedto represent string values, such as “walking”, “running”, “driving” etc.In such cases, 502 may be omitted.

At 504, an encoding context is selected for encoding a first value for apoint. The value may be an assigned attribute value or may be anattribute correction value, for example. The encoding context may beselected from a plurality of supported encoding contexts. For example, acontext store, such as context store 216 of an encoder, such as encoder202, as illustrated in FIG. 2A, may store a plurality of supportedencoding context for encoding attribute values or attribute correctionvalues for points of a point cloud. In some embodiments, an encodingcontext may be selected based on characteristics of a value to beencoded. For example, some encoding contexts may be optimized forencoding values with certain characteristics while other encodingcontexts may be optimized for encoding values with othercharacteristics.

In some embodiments, an encoding context may be selected based on aquantity or variety of symbols included in a value to be encoded. Forexample, values with fewer or less diverse symbols may be encoded usingarithmetic encoding techniques, while values with more symbols or morediverse symbols may be encoding using exponential Golomb encodingtechniques. In some embodiments, an encoding context may encode portionsof a value using more than one encoding technique. For example, in someembodiments, an encoding context may indicate that a portion of a valueis to be encoded using an arithmetic encoding technique and anotherportion of the value is to be encoded using a Golomb encoding technique.In some embodiments, an encoding context may indicate that a portion ofa value below a threshold is to be encoded using a first encodingtechnique, such as arithmetic encoding, whereas another portion of thevalue exceeding the threshold is to be encoded using another encodingtechnique, such as exponential Golomb encoding. In some embodiments, acontext store may store multiple encoding contexts, wherein eachencoding context is suited for values having particular characteristics.

At 506, a first value (or additional value) for the point may be encodedusing the encoding context selected at 504. At 508 it is determined ifthere are additional values for the point that are to be encoded. Ifthere are additional values for the point to be encoded, the additionalvalues may be encoded, at 506, using the same selected encodingtechnique that was selected at 504. For example, a point may have a“Red”, a “Green”, and a “Blue” color attribute. Because differencesbetween adjacent points in the R, G, B color space may be similar,attribute correction values for the Red attribute, Green attribute, andBlue attribute may be similar. Thus, in some embodiments, an encoder mayselect an encoding context for encoding attribute correction values fora first one of the color attributes, for example the Red attribute, andmay use the same encoding context for encoding attribute correctionvalues for the other color attributes, such as the Green attribute andthe Blue attribute.

At 510 encoded values, such as encoded assigned attribute values andencoded attribute correction values may be included in a compressedattribute information file. In some embodiments, the encoded values maybe included in the compressed attribute information file in accordancewith the evaluation order determined for the point cloud based on aminimum spanning tree. Thus a decoder may be able to determine whichencoded value goes with which attribute of which point based on theorder in which encoded values are included in a compressed attributeinformation file. Additionally, in some embodiments, data may beincluded in a compressed attribute information file indicatingrespective ones of the encoding contexts that were selected to encoderespective ones of the values for the points.

FIGS. 6A-B illustrate an example process for compressing spatialinformation of a point cloud, according to some embodiments.

At 602, an encoder receives a point cloud. The point cloud may be acaptured point cloud from one or more sensors or may be a generatedpoint cloud, such as a point cloud generated by a graphics application.For example, 604 illustrates points of an un-compressed point cloud.

At 606, the encoder sub-samples the received point cloud to generate asub-sampled point cloud. The sub-sampled point cloud may include fewerpoints than the received point cloud. For example, the received pointcloud may include hundreds of points, thousands of points, or millionsof points and the sub-sampled point cloud may include tens of points,hundreds of points or thousands of points. For example, 608 illustratessub-sampled points of a point cloud received at 602, for example asub-sampling of the points of the point cloud in 604.

In some embodiments, the encoder may encode and decode the sub-sampledpoint cloud to generate a representative sub-sampled point cloud thedecoder will encounter when decoding the compressed point cloud. In someembodiments, the encoder and decoder may execute a lossycompression/decompression algorithm to generate the representativesub-sampled point cloud. In some embodiments, spatial information forpoints of a sub-sampled point cloud may be quantized as part ofgenerating a representative sub-sampled point cloud. In someembodiments, an encoder may utilize lossless compression techniques andencoding and decoding the sub-sampled point cloud may be omitted. Forexample, when using lossless compression techniques the originalsub-sampled point cloud may be representative of a sub-sampled pointcloud the decoder will encounter because in lossless compression datamay not be lost during compression and decompression.

At 610, the encoder identifies subdivision locations between points ofthe sub-sampled point cloud according to configuration parametersselected for compression of the point cloud or according to fixedconfiguration parameters. The configuration parameters used by theencoder that are not fixed configuration parameters are communicated toan encoder by including values for the configuration parameters in acompressed point cloud. Thus, a decoder may determine the samesubdivision locations as the encoder evaluated based on subdivisionconfiguration parameters included in the compressed point cloud. Forexample, 612 illustrates identified sub-division locations betweenneighboring points of a sub-sampled point cloud.

At 614, the encoder determines for respective ones of the subdivisionlocations whether a point is to be included or not included at thesubdivision location in a decompressed point cloud. Data indicating thisdetermination is encoded in the compressed point cloud. In someembodiments, the data indicating this determination may be a single bitthat if “true” means a point is to be included and if “false” means apoint is not to be included. Additionally, an encoder may determine thata point that is to be included in a decompressed point cloud is to berelocated relative to the subdivision location in the decompressed pointcloud. For example 616, shows some points that are to be relocatedrelative to a subdivision location. For such points, the encoder mayfurther encode data indicating how to relocate the point relative to thesubdivision location. In some embodiments, location correctioninformation may be quantized and entropy encoded. In some embodiments,the location correction information may comprise delta X, delta Y,and/or delta Z values indicating how the point is to be relocatedrelative to the subdivision location. In other embodiments, the locationcorrection information may comprise a single scalar value whichcorresponds to the normal component of the location correctioninformation computed as follows:

ΔN=([X _(A) ,Y _(A) ,Z _(A)]−[X,Y,Z])·[Normal Vector]

In the above equation, delta N is a scalar value indicating locationcorrection information that is the difference between the relocated oradjusted point location relative to the subdivision location (e.g.[X_(A), Y_(A), Z_(A)]) and the original subdivision location (e.g. [X,Y, Z]). The cross product of this vector difference and the normalvector at the subdivision location results in the scalar value delta N.Because a decoder can determine, the normal vector at the subdivisionlocation, and can determine the coordinates of the subdivision location,e.g. [X, Y, Z], the decoder can also determine the coordinates of theadjusted location, e.g. [X_(A), Y_(A), Z_(A)], by solving the aboveequation for the adjusted location, which represents a relocatedlocation for a point relative to the subdivision location. In someembodiments, the location correction information may be furtherdecomposed into a normal component and one or more additional tangentialcomponents. In such an embodiment, the normal component, e.g. delta N,and the tangential component(s) may be quantized and encoded forinclusion in a compressed point cloud.

In some embodiments, an encoder may determine whether one or moreadditional points (in addition to points included at subdivisionlocations or points included at locations relocated relative tosubdivision locations) are to be included in a decompressed point cloud.For example, if the original point cloud has an irregular surface orshape such that subdivision locations between points in the sub-sampledpoint cloud do not adequately represent the irregular surface or shape,the encoder may determine to include one or more additional points inaddition to points determined to be included at subdivision locations orrelocated relative to subdivision locations in the decompressed pointcloud. Additionally, an encoder may determine whether one or moreadditional points are to be included in a decompressed point cloud basedon system constraints, such as a target bitrate, a target compressionratio, a quality target metric, etc. In some embodiments, a bit budgetmay change due to changing conditions such as network conditions,processor load, etc. In such embodiments, an encoder may adjust aquantity of additional points that are encoded to be included in adecompressed point cloud based on a changing bit budget. In someembodiments, an encoder may include additional points such that a bitbudget is consumed without being exceeded. For example, when a bitbudget is higher, an encoder may include more additional points toconsume the bit budget (and enhance quality) and when the bit budget isless, the encoder may include fewer additional points such that the bitbudget is consumed but not exceeded.

In some embodiments, an encoder may further determine whether additionalsubdivision iterations are to be performed. If so, the points determinedto be included, relocated, or additionally included in a decompressedpoint cloud are taken into account and the process reverts to 610 toidentify new subdivision locations of an updated sub-sampled point cloudthat includes the points determined to be included, relocated, oradditionally included in the decompressed point cloud. In someembodiments, a number of subdivision iterations to be performed (N) maybe a fixed or configurable parameter of an encoder. In some embodiments,different subdivision iteration values may be assigned to differentportions of a point cloud. For example, an encoder may take into accounta point of view from which the point cloud is being viewed and mayperform more subdivision iterations on points of the point cloud in theforeground of the point cloud as viewed from the point of view and fewersubdivision iterations on points in a background of the point cloud asviewed from the point of view.

At 618, the spatial information for the sub-sampled points of the pointcloud are encoded. Additionally, subdivision location inclusion andrelocation data is encoded. Additionally, any configurable parametersselected by the encoder or provided to the encoder from a user areencoded. The compressed point cloud may then be sent to a receivingentity as a compressed point cloud file, multiple compressed point cloudfiles, or may be packetized and communicated via multiple packets to areceiving entity, such as a decoder or a storage device. In someembodiments, a compressed point cloud may comprise both compressedspatial information and compressed attribute information. In otherembodiments, compressed spatial information and compressed attributeinformation may be included is separate compressed point cloud files.

FIG. 7 illustrates another example process for compressing spatialinformation of a point cloud, according to some embodiments.

In some embodiments, other spatial information compression techniquesother than the sub-sampling and prediction spatial information techniquedescribed in FIGS. 6A-B may be used. For example, a spatial encoder,such as spatial encoder 204, or a spatial decoder, such as spatialdecoder 222, may utilize other spatial information compressiontechniques, such as a K-D tree spatial information compressiontechnique. For example, compressing spatial information at 406 of FIG. 4may be performed using a sub-sampling and prediction technique similarto what is described in FIGS. 6A-B, may be performed using a K-D treespatial information compression technique similar to what is describedin FIG. 7, or may be performed using another suitable spatialinformation compression technique.

In a K-D tree spatial information compression technique, a point cloudcomprising spatial information may be received at 702. In someembodiments, the spatial information may have been previously quantizedor may further be quantized after being received. For example 718illustrates a captured point cloud that may be received at 702. Forsimplicity, 718 illustrates a point cloud in two dimensions. However, insome embodiments, a received point cloud may include points in 3-Dspace.

At 704, a K-dimensional tree or K-D tree is built using the spatialinformation of the received point cloud. In some embodiments, a K-D treemay be built by dividing a space, such as a 1-D, 2-D, or 3-D space of apoint cloud in half in a predetermined order. For example, a 3-D spacecomprising points of a point cloud may initially be divided in half viaa plane intersecting one of the three axis, such as the X-axis. Asubsequent division may then divide the resulting space along anotherone of the three axis, such as the Y-axis. Another division may thendivide the resulting space along another one of the axis, such as theZ-axis. Each time a division is performed a number of points included ina child cell created by the division may be recorded. In someembodiments, only a number of points in one child cell of two childcells resulting from a division may be recorded. This is because anumber of points included in the other child cell can be determined bysubtracting the number of points in the recorded child cell from a totalnumber of points in a parent cell prior to the division.

A K-D tree may include a sequence of number of points included in cellsresulting from sequential divisions of a space comprising points of apoint cloud. In some embodiments, building a K-D tree may comprisecontinuing to subdivide a space until only a single point is included ineach lowest level child cell. A K-D tree may be communicated as asequence of number of points in sequential cells resulting fromsequential divisions. A decoder may be configured with informationindicating the subdivision sequence followed by an encoder. For example,an encoder may follow a pre-defined division sequence until only asingle point remains in each lowest level child cell. Because thedecoder may know the division sequence that was followed to build theK-D tree and the number of points that resulted from each subdivision(which is communicated to the decoder as compressed spatial information)the decoder may be able to reconstruct the point cloud.

For example, 720 illustrates a simplified example of K-D compression ina two-dimensional space. An initial space includes seven points. Thismay be considered a first parent cell and a K-D tree may be encoded witha number of points “7” as a first number of the K-D tree indicating thatthere are seven total points in the K-D tree. A next step may be todivide the space along the X-axis resulting in two child cells, a leftchild cell with three points and a right child cell with four points.The K-D tree may include the number of points in the left child cell,for example “3” as a next number of the K-D tree. Recall that the numberof points in the right child cell can be determined based on subtractingthe number of points in the left child cell from the number of points inthe parent cell. A further step may be to divide the space an additionaltime along the Y-axis such that each of the left and right child cellsare divided in half into lower level child cells. Again, a number ofpoints included in the left lower-level child cells may be included in aK-D tree, for example “0” and “1”. A next step may then be to divide thenon-zero lower-level child cells along the X-axis and record the numberof points in each of the lower-level left child cells in a K-D tree.This process may continue until only a single point remains in a lowestlevel child cell. A decoder may utilize a reverse process to recreate apoint cloud based on receiving a sequence of point totals for each leftchild cell of a K-D tree.

At 706, an encoding context for encoding a number of points for a firstcell of the K-D tree, for example the parent cell comprising sevenpoints, is selected. In some embodiments, a context store may storehundreds or thousands of encoding contexts. In some embodiments, cellscomprising more points than a highest number of points encoding contextmay be encoded using the highest number point encoding context. In someembodiments, an encoding context may include arithmetic encoding, Golombexponential encoding, or a combination of the two. In some embodiments,other encoding techniques may be used. In some embodiments, anarithmetic encoding context may include probabilities for particularsymbols, wherein different arithmetic encoding contexts includedifferent symbol probabilities.

At 708, the number of points for the first cell is encoded according theselected encoding context.

At 710, an encoding context for encoding a child cell is selected basedon a number of points included in a parent cell. The encoding contextfor the child cell may be selected in a similar manner as for the parentcell at 706.

At 712, the number of points included in the child cell is encodedaccording the selected encoding context, selected at 710. At 714, it isdetermined if there are additional lower-level child cells to encode inthe K-D tree. If so, the process reverts to 710. If not, at 716, theencoded number of points in the parent cell and the child cells areincluded in a compressed spatial information file, such as a compressedpoint cloud. The encoded values are ordered in the compressed spatialinformation file such that the decoder may reconstruct the point cloudbased on the number of points of each parent and child cell and theorder in which the number of points of the respective cells are includedin the compressed spatial information file.

In some embodiments, the number of points in each cell may be determinedand subsequently encoded as a group at 716. Or, in some embodiments, anumber of points in a cell may be encoded subsequent to being determinedwithout waiting for all child cell point totals to be determined.

FIG. 8 illustrates an example process for decompressing compressedattribute information of a point cloud, according to some embodiments.

At 802, a decoder receives compressed attribute information for a pointcloud, and at 804, the decoder receives compressed spatial informationfor the point cloud. In some embodiments, the compressed attributeinformation and the compressed spatial information may be included inone or more common files or separate files.

At 806, the decoder decompresses the compressed spatial information. Thecompressed spatial information may have been compressed according to asub-sampling and prediction technique and the decoder may performsimilar sub-sampling, prediction, and prediction correction actions aswere performed at the encoder and further apply correction values to thepredicted point locations, to generate a non-compressed point cloud fromthe compressed spatial information. In some embodiments, the compressedspatial information may be compressed in a K-D tree format, and thedecoder may generate a decompressed point cloud based on an encoded K-Dtree included in the received spatial information. In some embodiments,the compressed spatial information may have been compressed using anOctree technique and an Octree decoding technique may be used togenerate decompressed spatial information for the point cloud. In someembodiments, other spatial information compression techniques may havebeen used and may be decompressed via the decoder.

At 808, the decoder may generate a minimum spanning tree, based on thedecompressed spatial information. For example, compressed spatialinformation and/or compressed attribute information may be received viaa encoded data interface of a decoder, such as encoded data interface226 of decoder 220 illustrated in FIG. 2B. A spatial decoder, such asspatial decoder 222, may decompress the compressed spatial information,and a minimum spanning tree generator, such as minimum spanning treegenerator 228, may generate a minimum spanning tree based on thedecompressed spatial information.

At 810, a prediction evaluator of a decoder, such as predictionevaluator 224 of decoder 220, may assign an attribute value to astarting point based on an assigned attribute value included in thecompressed attribute information. In some embodiments, the compressedattribute information may identify a point as a starting point to beused for generating the minimum spanning tree and for predictingattribute values of the points according to an evaluation order based onthe minimum spanning tree. The assigned attribute value or values forthe starting point may be included in decompressed attribute informationfor a decompressed point cloud.

At 812, the prediction evaluator of the decoder or another decodercomponent determines an evaluation order for at least the next pointsubsequent to the starting point that is to be evaluated. In someembodiments, an evaluation order may be determined for all or multipleones of the points, or in other embodiments, an evaluation order may bedetermined point by point as attribute values are determined for thepoints. The points may be evaluated in an order based on minimumdistances between successive points being evaluated. For example, aneighboring point at a shortest distance from a starting point ascompared to other neighboring points may be selected as a next point toevaluate subsequent to the starting point. In a similar manner, otherpoints may then be selected to be evaluated based on a shortest distancefrom a point that has most recently been evaluated. At 814, the nextpoint to evaluate is selected. In some embodiments 812 and 814 may beperformed together.

At 816, a prediction evaluator of a decoder determines the “K” nearestneighboring points to a point being evaluated. In some embodiments,neighboring points may only be included in the “K” nearest neighboringpoints if they already have assigned or predicted attribute values. Inother embodiments, neighboring points may be included in the “K” nearestneighboring points without regard to whether they have assigned oralready predicted attribute values. In such embodiments, an encoder mayfollow a similar rule as the decoder as to whether or not to includepoints without predicted values as neighboring points when identifyingthe “K” nearest neighboring points.

At 818, predicted attribute values are determined for one or moreattributes of the point being evaluated based on attribute values of the“K” nearest neighboring points and distances between the point beingevaluated and respective ones of the “K” nearest neighboring points. Insome embodiments, an inverse distance interpolation technique may beused to predict attribute values, wherein attribute values of pointscloser to a point being evaluated are weighted more heavily thanattribute values of points that are further away from the point beingevaluated. The attribute prediction technique used by a decoder may bethe same as an attribute prediction technique used by an encoder thatcompressed the attribute information.

At 820, a prediction evaluator of a decoder may apply an attributecorrection value to a predicted attribute value of a point to correctthe attribute value. The attribute correction value may cause theattribute value to match or nearly match an attribute value of anoriginal point cloud prior to compression. In some embodiments, in whicha point has more than one attribute, 818 and 820 may be repeated foreach attribute of the point. In some embodiments, some attributeinformation may be decompressed without decompressing all attributeinformation for a point cloud or a point. For example, a point mayinclude velocity attribute information and color attribute information.The velocity attribute information may be decoded without decoding thecolor attribute information and vice versa. In some embodiments, anapplication utilizing the compressed attribute information may indicatewhat attributes are to be decompressed for a point cloud.

At 822, it is determined if there are additional points to evaluate. Ifso, the process reverts to 814 and a next point to evaluate is selected.If there are not additional points to evaluate, at 824, decompressedattribute information is provided, for example as a decompressed pointcloud, wherein each point comprises spatial information and one or moreattributes.

Level of Detail Attribute Compression

In some circumstances, a number of bits needed to encode attributeinformation for a point cloud may make up a significant portion of a bitstream for the point cloud. For example, the attribute information maymake up a larger portion of the bit stream than is used to transmitcompressed spatial information for the point cloud.

In some embodiments, spatial information may be used to build ahierarchical Level of Detail (LOD) structure. The LOD structure may beused to compress attributes associated with a point cloud. The LODstructure may also enable advanced functionalities such asprogressive/view-dependent streaming and scalable rendering. Forexample, in some embodiments, compressed attribute information may besent (or decoded) for only a portion of the point cloud (e.g. a level ofdetail) without sending (or decoding) all of the attribute informationfor the whole point cloud.

FIG. 9 illustrates an example encoding process that generates ahierarchical LOD structure, according to some embodiments. For example,in some embodiments, an encoder such as encoder 202 may generatecompressed attribute information in a LOD structure using a similarprocess as shown in FIG. 9.

In some embodiments, geometry information (also referred to herein as“spatial information”) may be used to efficiently predict attributeinformation. For example, in FIG. 9 the compression of color informationis illustrated. However, a LOD structure may be applied to compressionof any type of attribute (e.g., reflectance, texture, modality, etc.)associated with points of a point cloud. Note that a pre-encoding stepwhich applies color space conversion or updates the data to make thedata better suited for compression may be performed depending on theattribute to be compressed.

In some embodiments, attribute information compression according to aLOD process proceeds as described below.

For example, let Geometry (G)={Point-P(0), P(1), . . . P(N−1)} bereconstructed point cloud positions generated by a spatial decoderincluded in an encoder (geometry decoder GD 902) after decoding acompressed geometry bit stream produced by a geometry encoder, alsoincluded in the encoder (geometry encoder GE 914), such as spatialencoder 204. For example, in some embodiments, an encoder such asencoder 202 may include both a geometry encoder, such as geometryencoder 914, and a geometry decoder, such as geometry decoder 914. Insome embodiments, a geometry encoder may be part of spatial encoder 214and a geometry decoder may be part of prediction/correction evaluator206. In some embodiments, a minimum spanning tree generator may beomitted, such as minimum spanning tree generator 210.

In some embodiments, the decompressed spatial information may describelocations of points in 3D space, such as X, Y, and Z coordinates of thepoints that make up mug 900. Note that spatial information may beavailable to both an encoder, such as encoder 202, and a decoder, suchas decoder 220. For example various techniques, such as K-D treecompression, octree compression, nearest neighbor prediction, etc., maybe used to compress and/or encode spatial information for mug 900 andthe spatial information may be sent to a decoder with, or in additionto, compressed attribute information for attributes of the points thatmake up a point cloud for mug 900.

In some embodiments, a deterministic re-ordering process may be appliedon both an encoder side (such as at encoder 202) and at a decoder side(such as at decoder 220) in order to organize points of a point cloud,such as the points that represent mug 900, in a set of Level of Details(LODs). For example, levels of detail may be generated by a level ofdetail generator 904, which may be included in a prediction/correctionevaluator of an encoder, such as prediction/correction evaluator 206 ofencoder 202. In some embodiments, a level of detail generator 904 may bea separate component of an encoder, such as encoder 202. For example,level of detail generator 904 may be a separate component of encoder202. Note that no additional information needs to be included in the bitstream to generate such LOD structures, except for the parameters of theLOD generation algorithm, For example, parameters that may be includedin a bit stream as parameters of the LOD generator algorithm mayinclude:

i. The maximum number of LODs to be generated denoted by “N” (e.g.,N=6),

ii. The initial sampling distance “D0” (e.g., D0=64), and

iii. The sampling distance update factor “f” (e.g., ½).

In some embodiments, the parameters N, D0 and f, may be provided by auser, such as an engineer configuring a compression process. In someembodiments the parameters N, D0 and f, may be determined automaticallyby an encoder/and or decoder using an optimization procedure, forexample. These parameters may be fixed or adaptive.

In some embodiments, LOD generation may proceed as follows:

-   -   a. Points of geometry G (e.g. the points of the point cloud        organized according to the spatial information), such as points        of mug 900, are marked as non-visited and a set of visited        points V is set to be empty.    -   b. The LOD generation process may then proceed iteratively. At        each iteration j, the level of detail for that refinement level,        e.g. LOD(j), may be generated as follows:        -   1. The sampling distance for the current LOD, denoted D(j)            may be set as follows:            -   a. If j=0, then D(j)=D0.            -   b. If j>0 and j<N, then D(j)=D(j−1)*f.            -   c. if j=N, then D(j)=0.        -   2. The LOD generation process iterates over all the points            of G.            -   a. At the point evaluation iteration i, a point P(i) is                evaluated,                -   i. if the point P(i) has been visited then it is                    ignored and the algorithm jumps to the next                    iteration (i+1), e.g. the next point P(i+1) is                    evaluated.                -   ii. Otherwise, the distance D(i, V), defined as the                    minimum distance from P(i) over all the points of V,                    is computed. Note that V is the list of points that                    have already been visited. If V is empty, the                    distance D(i, V) is set to 0, meaning that the                    distance from point P(i) to the visited points is                    zero because there are not any visited points in the                    set V. If the shortest distance from point P(i) to                    any of the already visited point, D(i, V), is                    strictly higher than a parameter D0, then the point                    is ignored and the LoD generation jumps to the                    iteration (i+1) and evaluates the next point P(i+1).                    Otherwise, P(i) is marked as a visited point and the                    point P(i) is added to the set of visited points V.            -   b. This process may be repeated until all the points of                geometry G are traversed.        -   3. The set of points added to V during the iteration j            describes the refinement level R(j).        -   4. The LOD(j) may be obtained by taking the union of all the            refinement levels R(0), R(1), . . . , R(j).

In some embodiments, the process described above, may be repeated untilall the LODs are generated or all the vertices have been visited.

In some embodiments, an encoder as described above may further include aquantization module (not shown) that quantizes geometry informationincluded in the “positions (x,y,z) being provided to the geometryencoder 914. Furthermore, in some embodiments, an encoder as describedabove may additionally include a module that removes duplicated pointssubsequent to quantization and before the geometry encoder 914.

In some embodiments, quantization may further be applied to compressedattribute information, such as attribute correction values and/or one ormore attribute value starting points. For example quantization isperformed at 910 to attribute correction values determined byinterpolation-based prediction module 908. Quantization techniques mayinclude uniform quantization, uniform quantization with a dead zone,non-uniform/non-linear quantization, trellis quantization, or othersuitable quantization techniques.

FIG. 10 illustrates an example process for determining points to beincluded at different refinement layers of a level of detail (LOD)structure, according to some embodiments.

At 1002 an encoder (or a decoder) receives or determines level of detailparameters to use in determining the level of detail hierarchy for thepoint cloud. At the same time, or before or after, receiving the levelof detail parameters, at 1004 the encoder (or decoder) may receivecompressed spatial information for the point cloud and at 1006, theencoder (or decoder) may determine decompressed spatial information forthe points of the point cloud. In embodiments that utilize lossycompression techniques for the compression of the spatial information,the compressed spatial information may be compressed at the encoder andmay also be decompressed at the encoder at 1006 to generate arepresentative sample of the geometry information that will beencountered at a decoder. In some embodiments that utilize losslesscompression of spatial information, 1004 and 1106 may be omitted on theencoder side.

At 1008 the level of detail structure generator (which may be on anencoder side or a decoder side) marks all the points of the point cloudas “non-visited points.”

At 1010 the level of detail structure generator also sets a directory ofvisited point “V” to be empty.

At 1012 a sampling distance D(j) is determined for a current level ofrefinement being evaluated, R(j). If the level of refinement is thecoarsest level of refinement, where j=0, the sampling distance D(j) isset to be equal to D0, e.g. the initial sampling distance, which wasreceived or determined at 1002. If j is greater than 0, but less than N,then D(j) is set to equal to D(j−1)*f. Note that “N” is the total numberof level of details that are to be determined. Also note that “f” is asampling update distance factor which is set to be less than one (e.g.½). Also, note that D(j−1) is the sampling distance used in thepreceding level of refinement. For example, when f is ½, the samplingdistance D(j−1) is cut in half for a subsequent level of refinement,such that D(j) is one half the length of D(j−1). Also, note that a levelof detail (LOD(j)) is the union of a current level of refinement and alllower levels of refinement. Thus, a first level of detail (LOD(0)) mayinclude all the points included in level of refinement R(0). Asubsequent level of detail (LOD(1) may include all of the pointsincluded in the previous level of detail and additionally all the pointsincluded in the subsequent level of refinement R(1). In this way pointsmay be added to a previous level of detail for each subsequent level ofdetail until a level of detail “N” is reached that includes all of thepoints of the point cloud.

To determine the points of the point cloud to be included in a currentlevel of detail which is being determined, a point P(i) is selected tobe evaluated at 1014, where “i” is a current one of the points of thepoint cloud that is being evaluated. For example if a point cloudincludes a million points, “i” could range from 0 to 1,000,000.

At 1016, it is determined if the point currently being evaluated, P(i),has already been marked as a visited point. If P(i) is marked as alreadyvisited, then at 1018 P(i) is ignored and the process moves on toevaluate the next point P(i+1), which then becomes the point currentlybeing evaluated P(i). The process then reverts back to 1014.

If it is determined at 1016, that point P(i) has not already been markedas a visited point, at 1020 a distance D(i) is computed for the pointP(i), where D(i) is the shortest distance between point P(i) and any ofthe already visited points included in directory V. If there are not anypoints included in directory V, e.g. the directory V is empty, then D(i)is set to zero.

At 1022, it is determined whether the distance D(i) for point P(i) isgreater than the initial sampling distance D0. If so, at 1018 point P(i)is ignored and the process moves on to the next point P(i+1) and revertsto 1014.

If point P(i) is not already marked as visited and the distance D(i),which is a minimum distance between point P(i) and the set of pointsincluded in V, is less than the initial sampling distance D0, then at1024 point P(i) is marked as visited and added to a set of visitedpoints V for the current level of refinement R(j).

At 1026, it is determined if there are additional levels of refinementto determine for the point cloud. For example, if j<N, then there areadditional levels of refinement to determine, where N is a LOD parameterthat may be communicated between an encoder and decoder. If there arenot additional levels of refinement to determine, the process stops at1028. If there are additional levels of refinement to determine theprocess moves on to the next level of refinement at 1030, and thenproceeds to evaluate the point cloud for the next level of refinement at1012.

Once the levels of refinement have been determined, the levels ofrefinement may be used generate the LOD structure, where each subsequentLOD level includes all the points of a previous LOD level plus anypoints determined to be included in an additional level of refinement.Because the process for determining an LOD structure is known by theencoder and decoder, a decoder, given the same LOD parameters as used atan encoder, can recreate the same LOD structure at the decoder as wasgenerated at the encoder.

Example Level of Detail Hierarchy

FIG. 11A illustrates an example LOD, according to some embodiments. Notethat the LOD generation process may generate uniformly sampledapproximations (or levels of detail) of the original point cloud, thatget refined as more and more points are included. Such a feature makesit particularly adapted for progressive/view-dependent transmission andscalable rendering. For example, 1104 may include more detail than 1102,and 1106 may include more detail than 1104. Also, 1108 may include moredetail than 1102, 1104, and 1106.

The hierarchical LOD structure may be used to build an attributeprediction strategy. For example, in some embodiments the points may beencoded in the same order as they were visited during the LOD generationphase. Attributes of each point may be predicted by using the K-nearestneighbors that have been previously encoded. In some embodiments, “K” isa parameter that may be defined by the user or may be determined byusing an optimization strategy. “K” may be static or adaptive. In thelatter case where “K” is adaptive, extra information describing theparameter may be included in the bit stream.

In some embodiments, different prediction strategies may be used. Forexample, one of the following interpolation strategies may be used, aswell as combinations of the following interpolation strategies, or anencoder/decoder may adaptively switch between the differentinterpolation strategies. The different interpolation strategies mayinclude interpolation strategies such as: inverse-distanceinterpolation, barycentric interpolation, natural neighborinterpolation, moving least squares interpolation, or other suitableinterpolation techniques. For example, interpolation based predictionmay be performed at an interpolation-based prediction module 908included in a prediction/correction value evaluator of an encoder, suchas prediction/correction value evaluator 206 of encoder 202. Also,interpolation based prediction may be performed at aninterpolation-based prediction module 908 included in a predictionevaluator of a decoder, such as prediction evaluator 224 of decoder 220.In some embodiments, a color space may also be converted, at color spaceconversion module 906, prior to performing interpolation basedprediction. In some embodiments, a color space conversion module 906 maybe included in an encoder, such as encoder 202. In some embodiments, adecoder may further included a module to convert a converted colorspace, back to an original color space.

In some embodiments, quantization may further be applied to attributeinformation. For example quantization may performed at quantizationmodule 910. In some embodiments, a encoder, such as encoder 202, mayfurther include a quantization module 910. Quantization techniquesemployed by a quantization module 910 may include uniform quantization,uniform quantization with a dead zone, non-uniform/non-linearquantization, trellis quantization, or other suitable quantizationtechniques.

In some embodiments, LOD attribute compression may be used to compressdynamic point clouds as follows:

-   -   a. Let FC be the current point cloud frame and RF be the        reference point cloud.    -   b. Let M be the motion field that deforms RF to take the shape        of FC.        -   i. M may be computed on the decoder side and in this case            information may not be encoded in the bit stream.        -   ii. M may be computed by the encoder and explicitly encoded            in the bit stream            -   1. M may be encoded by applying a hierarchical                compression technique as described herein to the motion                vectors associated with each point of RF (e.g. the                motion of RF may be considered as an extra attribute).            -   2. M may be encoded as a skeleton/skinning-based model                with associated local and global transforms.            -   3. M may be encoded as a motion field defined based on                an octree structure, which is adaptively refined to                adapt to motion field complexity.            -   4. M may be described by using any suitable animation                technique such as key-frame-based animations, morphing                techniques, free-form deformations, key-point-based                deformation, etc.        -   iii. Let RF′ be the point cloud obtained after applying the            motion field M to RF. The points of RF′ may be then used in            the attribute prediction strategy by considering not only            the “K” nearest neighbor points of FC but also those of RF′.

Furthermore, attribute correction values may be determined based oncomparing the interpolation-based prediction values determined atinterpolation-based prediction module 908 to original non-compressedattribute values. The attribute correction values may further bequantized at quantization module 910 and the quantitated attributecorrection values, encoded spatial information (output from the geometryencoder 902) and any configuration parameters used in the prediction maybe encoded at arithmetic encoding module 912. In some embodiments, thearithmetic encoding module, may use a context adaptive arithmeticencoding technique. The compressed point cloud may then be provided to adecoder, such as decoder 220, and the decoder may determine similarlevels of detail and perform interpolation based prediction to recreatethe original point cloud based on the quantized attribute correctionvalues, encoded spatial information (output from the geometry encoder902) and the configuration parameters used in the prediction at theencoder.

FIG. 11B illustrates an example compressed point cloud file comprisingLODs, according to some embodiments. Level of detail attributeinformation file 1150 includes configuration information 1152, pointcloud data 1154, and level of detail point attribute correction values1156. In some embodiments, level of detail attribute information file1150 may be communicated in parts via multiple packets. In someembodiments, not all of the sections shown in the level of detailattribute information file 1150 may be included in each packettransmitting compressed attribute information. In some embodiments, alevel of detail attribute information file, such as level of detailattribute information file 1150, may be stored in a storage device, suchas a server that implements an encoder or decoder, or other computingdevice.

FIG. 12A illustrates a method of encoding attribute information of apoint cloud, according to some embodiments.

At 1202, a point cloud is received by an encoder. The point cloud may becaptured, for example by one or more sensors, or may be generated, forexample in software.

At 1204, spatial or geometry information of the point cloud is encodedas described herein. For example, the spatial information may be encodedusing K-D trees, Octrees, a neighbor prediction strategy, or othersuitable technique to encoded the spatial information.

At 1206, one or more level of details are generated, as describedherein. For example, the levels of detail may be generated using asimilar process as shown in FIG. 10. Note that in some embodiments, thespatial information encoded or compressed at 1204 may be de-coded ordecompressed to generate a representative decompressed point cloudgeometry that a decoder would encounter. This representativedecompressed point cloud geometry may then be used to generate a LODstructure as further described in FIG. 10.

At 1208, an interpolation based prediction is performed to predictattribute values for the attributes of the points of the point cloud. At1210, attribute correction values are determined based on comparing thepredicted attribute values to original attribute values. For example, insome embodiments, an interpolation based prediction may be performed foreach level of detail to determine predicted attribute values for pointsincluded in the respective levels of detail. These predicted attributevalues may then be compared to attribute values of the original pointcloud prior to compression to determine attribute correction values forthe points of the respective levels of detail. For example, aninterpolation based prediction process as described in FIG. 1B, FIGS.4-5, and FIG. 8 may be used to determine predicted attribute values forvarious levels of detail. In some embodiments, attribute correctionvalues may be determined for multiple levels of detail of a LODstructure. For example a first set of attribute correction values may bedetermined for points included in a first level of detail and additionalsets of attribute correction values may be determined for pointsincluded in other levels of detail.

At 1212, an update operation may optionally be applied that affects theattribute correction values determined at 1210. Performance of theupdate operation is discussed in more detail below in FIG. 12C.

At 1214, attribute correction values, LOD parameters, encoded spatialinformation (output from the geometry encoder) and any configurationparameters used in the prediction are encoded, as described herein.

In some embodiments, the attribute information encoded at 1214 mayinclude attribute information for multiple or all levels of detail ofthe point cloud, or may include attribute information for a single levelof detail or fewer than all levels of detail of the point cloud. In someembodiments, level of detail attribute information may be sequentiallyencoded by an encoder. For example, an encoder may make available afirst level of detail before encoding attribute information for one ormore additional levels of detail.

In some embodiments, an encoder may further encode one or moreconfiguration parameters to be sent to a decoder, such as any of theconfiguration parameters shown in configuration information 1152 ofcompressed attribute information file 1150. For example, in someembodiments, an encoder may encode a number of levels of detail that areto be encoded for a point cloud. The encoder may also encode a samplingdistance update factor, wherein the sampling distance is used todetermine which points are to be included in a given level of detail.

FIG. 12B illustrates a method of decoding attribute information of apoint cloud, according to some embodiments.

At 1252, compressed attribute information for a point cloud is receivedat a decoder. Also, at 1254 spatial information for the point cloud isreceived at the decoder. In some embodiments, the spatial informationmay be compressed or encoded using various techniques, such as a K-Dtree, Octree, neighbor prediction, etc. and the decoder may decompressand/or decode the received spatial information at 1254.

At 1256, the decoder determines which level of detail of a number oflevels of detail to decompress/decode. The selected level of detail todecompress/decode may be determined based on a viewing mode of the pointcloud. For example, a point cloud being viewed in a preview mode mayrequire a lower level of detail to be determined than a point cloudbeing viewed in a full view mode. Also, a location of a point cloud in aview being rendered may be used to determine a level of detail todecompress/decode. For example, a point cloud may represent an objectsuch as the coffee mug shown in FIG. 9A. If the coffee mug is in aforeground of a view being rendered a higher level of detail may bedetermined for the coffee mug. However, if the coffee mug is in thebackground of a view being rendered, a lower level of detail may bedetermined for the coffee mug. In some embodiments, a level of detail todetermine for a point cloud may be determined based on a data budgetallocated for the point cloud.

At 1258 points included in the first level of detail (or next level ofdetail) being determined may be determined as described herein. For thepoints of the level of detail being evaluated, attribute values of thepoints may be predicted based on an inverse distance weightedinterpolation based on the k-nearest neighbors to each point beingevaluated, where k may be a fixed or adjustable parameter.

At 1260, in some embodiments, an update operation may be performed onthe predicted attribute values as described in more detail in FIG. 12F.

At 1262, attribute correction values included in the compressedattribute information for the point cloud may be decoded for the currentlevel of detail being evaluated and may be applied to correct theattribute values predicted at 1258 or the updated predicted attributevalues determined at 1260.

At 1264, the corrected attribute values determined at 1262 may beassigned as attributes to the points of the first level of detail (orthe current level of detail being evaluated). In some embodiments, theattribute values determined for subsequent levels of details may beassigned to points included in the subsequent levels of detail whileattribute values already determined for previous levels of detail areretained by the respective points of the previous level(s) of detail. Insome embodiments, new attribute values may be determined for sequentiallevels of detail.

In some embodiments, the spatial information received at 1254 mayinclude spatial information for multiple or all levels of detail of thepoint cloud, or may include spatial information for a single level ofdetail or fewer than all levels of detail of the point cloud. In someembodiments, level of detail attribute information may be sequentiallyreceived by a decoder. For example, a decoder may receive a first levelof detail and generate attribute values for points of the first level ofdetail before receiving attribute information for one or more additionallevels of detail.

At 1266 it is determined if there are additional levels of detail todecode. If so, the process returns to 1258 and is repeated for the nextlevel of detail to decode. If not the process is stopped at 1267, butmay resume at 1256 in response to input affecting the number of levelsof detail to determine, such as change in view of a point cloud or azoom operation being applied to a point cloud being viewed, as a fewexamples of an input affecting the levels of detail to be determined.

In some embodiments the spatial information described above may beencoded and decoded via a geometry encoder and arithmetic encoder, suchas geometry encoder 202 and arithmetic encoder 212 described above inregard to FIG. 2. In some embodiments, a geometry encoder, such asgeometry encoder 202 may utilize an octree compression technique andarithmetic encoder 212 may be a binary arithmetic encoder as describedin more detail below.

The use of a binary arithmetic encoder as described below reduces thecomputational complexity of encoding octree occupancy symbols ascompared to a multi-symbol codec with an alphabet of 256 symbols (e.g. 8sub-cubes per cube, and each sub-cube occupied or un-occupied2{circumflex over ( )}8=256). Also the use of context selection based onmost probable neighbor configurations may reduce a search for neighborconfigurations, as compared to searching all possible neighborconfigurations. For example, the encoder may keep track of 10 encodingcontexts which correspond to the 10 neighborhood configurations 1268,1270, 1272, 12712, 1276, 1278, 1280, 1282, 1284, and 1286 shown in FIG.12C as opposed to all possible neighborhood configurations.

In some embodiments, an arithmetic encoder, such as arithmetic encoder212, may use a binary arithmetic codec to encode the 256-value occupancysymbols. This may be less complex and more hardware friendly in terms ofimplementation as compared to a multi-symbol arithmetic codec.Additionally, an arithmetic encoder 212 and/or geometry encoder 202 mayutilizes a look-ahead procedure to compute the 6-neighbors used forarithmetic context selection, which may be less complex than a linearsearch and may involve a constant number of operations (as compared to alinear search which may involve varying numbers of operations).Additionally, the arithmetic encoder 212 and/or geometry encoder 202 mayutilize a context selection procedure, which reduces the number ofencoding contexts. In some embodiments, a binary arithmetic codec,look-ahead procedure, and context selection procedure may be implementedtogether or independently.

Binary Arithmetic Encoding

In some embodiments, to encode spatial information, occupancyinformation per cube is encoded as an 8-bit value that may have a valuebetween 0-255. To perform efficient encoding/decoding of such non-binaryvalues, typically a multi-symbol arithmetic encoder/decoder would beused, which is computationally complex and less hardware friendly toimplement when compared to a binary arithmetic encoder/decoder. However,direct use of a conventional binary arithmetic encoder/decoder on such avalue on the other hand, e.g. encoding each bit independently, may notbe as efficient. However, in order, to efficiently encode the non-binaryoccupancy values with a binary arithmetic encoder an adaptive look uptable (A-LUT), which keeps track of the N (e.g., 32) most frequentoccupancy symbols, may be used along with a cache which keeps track ofthe last different observed M (e.g., 16) occupancy symbols.

The values for the number of last different observed occupancy symbols Mto track and the number of the most frequent occupancy symbols N totrack may be defined by a user, such as an engineer customizing theencoding technique for a particular application, or may be chosen basedon an offline statistical analysis of encoding sessions. The choice ofthe values of M and N may be based on a compromise between:

-   -   Encoding efficiency,    -   Computational complexity, and    -   Memory requirements.

In some embodiments, the algorithm proceeds as follows:

-   -   The adaptive look-up table (A-LUT) is initialized with N symbols        provided by the user (e.g. engineer) or computed offline based        on the statistics of a similar class of point clouds.    -   The cache is initialized with M symbols provided by the user        (e.g. engineer) or computed offline based on the statistics of a        similar class of point clouds.    -   Every time an occupancy symbol S is encoded the following steps        are applied        -   1. A binary information indicating whether S is in the A-LUT            or not is encoded.        -   2. If S is in the A-LUT, the index of S in the A-LUT is            encoded by using a binary arithmetic encoder            -   Let (b1, b2, b3, b4, b5) be the five bits of the binary                representation of the index of S in the A-LUT. Let b1 be                the least significant bit and b5 the most significant                bit.            -   Three approaches as described below to encode the index                of S may be used, for example by using either 31, 9, or                5 adaptive binary arithmetic contexts as shown below                -   31 Contexts                -    First encode b5 of the index of S with a first                    context (call it context 0), when encoding the most                    significant bit (the first bit to be encoded) there                    is not any information that can be used from the                    encoding of other bits, that is why the context is                    referred to as context zero. Then when encoding b4                    (the second bit to be encoded), there are two                    additional contexts that may be used call them                    context 1 (if b5=0) and context 2 (if b5=1). When                    this approach is taken all the way out to b1, there                    are 31 resulting contexts as shown in the diagram                    below, context 0-30. This approach exhaustively uses                    each bit that is encoded to select an adaptive                    context for encoding the next bit. For example, see                    FIG. 12E.                -   9 Contexts                -   Keep in mind that the index values of the adaptive                    look-up table ALUT are assigned based on how                    frequently the symbol S has appeared. Thus the most                    frequent symbol S in the ALUT would have an index                    value of 0 meaning that all of the bits of the index                    value for the most frequent symbol S are zero. For                    example, the smaller the binary value, the more                    frequently the symbol has appeared. To encode nine                    contexts, for b4 and b5, which are the most                    significant bits, if they are 1 s the index value                    must be comparatively large. For example if b5=1                    then the index value is at least 16 or higher, or if                    b4=1 the index value is at least 8 or higher. So                    when encoding 9 contexts, the focus is placed on the                    first 7 index entries, for example 1-7. For these 7                    index entries adaptive encoding contexts are used.                    However for index entries with values greater than 7                    the same context is used, for example a static                    binary encoder. Thus, if b5=1 or b4=1, then the same                    context is used to encode the index value. If not,                    then one of the adaptive contexts 1-7 is used.                    Because there is a context 0 for b5, 7 adaptive                    contexts, and a common context for entries strictly                    greater than 8, there are nine total contexts. This                    simplifies encoding and reduces the number of                    contexts to be communicated as compared to using all                    31 contexts as shown above.                -   5 contexts                -   To encode an index value using 5 contexts, determine                    if b5=1. If b5=1 then use a static binary context to                    encode all the bits of the index value from b4 to                    b1. If b5 does not equal 1, then encode b4 of the                    index value and see if b4 is equal to 1 or 0. If                    b4=1, which means the index value is higher than 8,                    then again use the static binary context to encode                    the bits b3 to b1. This reasoning then repeats, so                    that if b3=1, the static binary context is used to                    encode bits b2 to b1, and if b2=1 the static binary                    context is used to encode bit 1. However, if bits                    b5, b4, and b3 are equal to zero, then an adaptive                    binary context is selected to encode bit 2 and bit 1                    of the index value.        -   3. If S is not in the A-LUT, then            -   A binary information indicating whether S is in the                cache or not is encoded.            -   If S is in the cache, then the binary representation of                its index is encoded by using a binary arithmetic                encoder                -   In some embodiments, the binary representation of                    the index is encoded by using a single static binary                    context to encode each bit, bit by bit. The bit                    values are then shifted over by one, where the least                    significant bit becomes the next more significant                    bit.            -   Otherwise, if S is not in the cache, then the binary                representation of S is encoded by using a binary                arithmetic encoder                -   In some embodiments, the binary representation of S                    is encoded by using a single adaptive binary                    context. It is known that the index have a value                    between 0 and 255, which means it is encoded on 8                    bits. The bits are shifted so that the least                    significant bit becomes the next more significant                    bit, and a same adaptive context is used to encode                    all of the remaining bits.            -   The symbol S is added to the cache and the oldest symbol                in the cache is evicted.        -   4. The number of occurrences of the symbol S in A-LUT is            incremented by one.        -   5. The list of the N most frequent symbols in the A-LUT is            re-computed periodically            -   Approach 1: If the number of symbols encoded so far                reaches a user-defined threshold (e.g., 64 or 128), then                the list of the N most frequent symbols in the A-LUT is                re-computed.            -   Approach 2: Adapts the update cycle to the number of                symbols encoded. The idea is to update the probabilities                fast in the beginning and exponentially increase the                update cycle with the number of symbols:                -   The update cycle updateCycle is initialized to a low                    number NO (e.g. 16).                -   Every time the number of symbols reaches the update                    cycle                -    the list of the N most frequent symbols in the                    A-LUT is re-computed                -    The update cycle is updated as follows:                    _updateCycle=min(alpha*updateCycle, _maxUpdateCycle)                -   _alpha (e.g., 5/4) and maxUpdateCycle (e.g., 1024)                    are two user-defined parameters, which may control                    the speed of the exponential growth and the maximum                    update cycle value.        -   6. At the start of each level of the octree subdivision, the            occurrences of all symbols are reset to zero. The            occurrences of the N most frequent symbols are set to 1.        -   7. When the occurrence of a symbol reaches a user-defined            maximum number (e.g., max_Occurence=1024), the occurrences            of all the symbols are divided by 2 to keep the occurrences            within a user-defined range.

In some embodiments, a ring-buffer is used to keep track of the elementsin the cache. The element to be evicted from the cache corresponds tothe position index( )=(_last++) % CacheSize, where _last is a counterinitialized to 0 and incremented every time a symbol is added to thecache. In some embodiments, the cache could also be implemented with anordered list, which would guarantee that every time the oldest symbol isevicted.

2. Look-ahead to determine neighbors

In some embodiments, at each level of subdivision of the octree, cubesof the same size are subdivided and an occupancy code for each one isencoded.

-   -   For subdivision level 0, there may be a single cube of        (2^(C),2^(C),2^(C)) without any neighbors.    -   For subdivision level 1, there may be up to 8 cubes of dimension        (2^(C−1),2^(C−1),2^(C−1)) each.    -   For subdivision level L, there may be up to 8^(L) cubes of        dimension (2^(C−L),2^(C−L),2^(C−L)) each.

In some embodiments, at each level L, a set of non-overlappinglook-ahead cubes of dimension (2H−C+L,2H−C+L,2H−C+L) each may bedefined, as shown in FIG. 12D. Note that the look-ahead cube can fit23×H cubes of size (2^(C−L),2^(C−L),2^(C−L)).

-   -   At each level L, the cubes contained in each look-ahead cube are        encoded without referencing cubes in other look-ahead cubes.        -   During the look-ahead phase, the cubes of dimension            (2^(C−L),2^(C−L),2^(C−L)) in the current look-ahead cube are            extracted from the FIFO and a look-up table that describes            for each (2^(C−L),2^(C−L),2^(C−L)) region of the current            look-ahead cube whether it is occupied or empty is filled.        -   Once, the look-up table is filled, the encode phase for the            extracted cubes begins. Here, the occupancy information for            the 6 neighbors is obtained by fetching the information            directly from the look up table.        -   For cubes on the boundary of the look-ahead cube, the            neighbors located outside are assumed to be empty.            -   Another alternative could consist in filling the values                of the outside neighbors based on extrapolation methods.        -   Efficient implementation could be achieved by            -   Storing the occupancy information of each group of 8                neighboring (2^(C−L),2^(C−L),2^(C−L)) regions on one                byte            -   Storing the occupancy bytes in a Z-order to maximize                memory cache hits

3. Context Selection

In some embodiments, to reduce the number of encoding contexts (NC) to aalower number of contexts (e.g., reduced from 10 to 6), a separatecontext is assigned to each of the (NC-1) most probable neighborhoodconfigurations, and the contexts corresponding to the least probableneighborhood configurations are made to share the same context(s). Thisis done as follows:

-   -   Before starting the encoding process, initialize the occurrences        of the 10 neighborhood configurations (e.g. the 10        configurations shown in FIG. 12C):        -   Set all 10 occurrences to 0        -   Set the occurrences based on offline/online statistics or            based on user-provided information.    -   At the beginning of each subdivision level of the octree:        -   Determine the (NC-1) most probable neighborhood            configurations based on the statistics collected during the            encoding of the previous subdivision level.        -   Compute a look-up table NLUT, which maps the indexes of the            (NC-1) most probable neighborhood configurations to the            numbers 0, 1, . . . , (NC-2) and maps the indexes of the            remaining configurations to NC-1.        -   Initialize the occurrences of the 10 neighborhood            configurations to 0.            -   During the encoding:                -   Increment the occurrence of a neighborhood                    configuration by one each time such a configuration                    is encountered.                -   Use the look-up table NLUT[ ] to determine the                    context to use to encode the current occupancy                    values based on the neighborhood configuration                    index.

FIG. 12F illustrates an example octree compression technique using abinary arithmetic encoder, cache, and look-ahead table, according tosome embodiments. For example, FIG. 12F illustrates an example of theprocesses as described above. At 1288 occupancy symbols for a level ofan octree of a point cloud are determined. At 1290 an adaptivelook-ahead table with “N” symbols is initialized. At 1292, a cache with“M” symbols is initialized. At 1294, symbols for the current octreelevel are encoded using the techniques described above. At 1296, it isdetermined if additional octree levels are to be encoded. If so, theprocess continues at 1288 for the next octree level. If not, the processends at 1298 and the encoded spatial information for the point cloud ismade available for use, such as being sent to a recipient or beingstored.

Lifting Schemes for Level of Detail Compression and Decompression

In some embodiments, lifting schemes may be applied to point clouds. Forexample, as described below, a lifting scheme may be applied toirregular points. This is in contrast to other types of lifting schemesthat may be applied to images having regular points in a plane. In alifting scheme, for points in a current level of detail nearest pointsin a lower level of detail are found. These nearest points in the lowerlevel of detail are used to predict attribute values for points in ahigher level of detail. Conceptually, a graph could be made showing howpoints in lower levels of detail are used to determine attribute valuesof points in higher levels of detail. In such a conceptual view, edgescould be assigned to the graph between levels of detail, wherein thereis an edge between a point in a higher level of detail and each point inthe lower level of detail that forms a basis for the prediction of theattribute of the point at the higher level of detail. As described inmore detail below, a weight could be assigned to each of these edgesindicating a relative influence. The weight may represent an influenceattribute value of the point in the lower level of detail has on theattribute value of the points in the higher level of detail. Also,multiple edges may make a path through the levels of detail and weightsmay be assigned to the paths. In some embodiments, the influence of apath may be defined by the sum of the weights of the edges of the path.For example, equation 1 below represents such a weighting of a path.

In a lifting scheme, low influence points may be highly quantized andhigh influence points may be quantized less. In some embodiments, abalance may be reached between quality of a reconstructed point cloudand efficiency, wherein more quantization increases compressionefficiency and less quantization increases quality. In some embodiments,all paths may not be evaluated. For example, some paths with littleinfluence may not be evaluated. Also, an update operator may smoothresidual differences, e.g. attribute correction values, in order toincrease compression efficiency while taking into account relativeinfluence or importance of points when smoothing the residualdifferences.

FIG. 13A illustrates a direct transformation that may be applied at anencoder to encode attribute information of a point could, according tosome embodiments.

In some embodiments, an encoder may utilize a direct transformation asillustrated in FIG. 13A in order to determine attribute correctionvalues that are encoded as part of a compressed point cloud. Forexample, in some embodiments a direct transformation may be utilized todetermine attribute values as described in FIG. 12A at 1208 and to applyan update operation as described in FIG. 12A at 1212.

In some embodiments, a direct transform may receive attribute signalsfor attributes associated with points of a point cloud that is to becompressed. For example, the attributes may include color values, suchas RGB colors, or other attribute values of points in a point cloud thatis to be compressed. The geometry of the points of the point cloud to becompressed may also be known by the direct transform that receives theattribute signals. At 1302, the direct transform may include a splitoperator that splits the attribute signals 1310 for a first (or next)level of detail. For example, for a particular level of detail, such asLOD(N), comprising X number of points, a sub-sample of the attributes ofthe points, e.g. a sample comprising Y points, may comprise attributevalues for a smaller number of points than X. Said another way, thesplit operator may take as an input attributes associated with aparticular level of detail and generate a low resolution sample 1304 anda high resolution sample 1306. It should be noted that a LOD structuremay be partitioned into refinement levels, wherein subsequent levels ofrefinement include attributes for more points than underlying levels ofrefinement. A particular level of detail as described below is obtainedby taking the union of all lower level of detail refinements. Forexample, the level of detail j is obtained by taking the union of allrefinement levels R(0), R(1), . . . , R(j). It should also be noted, asdescribed above, that a compressed point cloud may have a total numberof levels of detail N, wherein R(0) is the least refinement level ofdetail and R(N) is the highest refinement level of detail for thecompressed point cloud.

At 1308, a prediction for the attribute values of the points notincluded in the low resolution sample 1304 is predicted based on thepoints included in the low resolution sample. For example, based on aninverse distance interpolation prediction technique or any of the otherprediction techniques described above. At 1312, a difference between thepredicted attribute values for the points left out of low resolutionsample 1304 is compared to the actual attribute values of the pointsleft out of the low resolution sample 1304. The comparison determinesdifferences, for respective points, between a predicted attribute valueand an actual attribute value. These differences (D(N)) are then encodedas attribute correction values for the attributes of the points includedin the particular level of detail that are not encoded in the lowresolution sample. For example, for the highest level of detail N, theattribute correction values D(N) may be encoded such that attributevalues not included in a low resolution sample of LOD(N) may bepredicted and corrected using the attribute correction values D(N).Because at the highest level of detail, the attribute correction valuesare not used to determine attribute values of other even higher levelsof detail (because for the highest level of detail, N, there are not anyhigher levels of detail), an update operation to account for relativeimportance of these attribute correction values may not be performed. Assuch, the differences D(N) may be used to encode attribute correctionvalues for LOD(N).

In addition, the direct transform may be applied for subsequent lowerlevels of detail, such as LOD(N−1). However, before applying the directtransform for the subsequent level of detail, an update operation may beperformed in order to determine the relative importance of attributevalues for points of the lower level of detail on attribute values ofone or more upper levels of detail. For example, update operation 1314may determine relative importances of attribute values of attributes forpoints included in lower levels of detail on higher levels of detail,such as for attributes of points included in L(N). The update operatormay also smooth the attributes values to improve compression efficiencyof attribute correction values for subsequent levels of detail takinginto account the relative importance of the respective attribute values,wherein the smoothing operation is performed such that attribute valuesthat have a larger impact on subsequent levels of detail are modifiedless than points that have a lesser impact on subsequent levels ofdetail. Several approaches for performing the update operation aredescribed in more detail below. The updated lower resolution sample oflevel of detail L′(N) is then fed to another split operator and theprocess repeats for a subsequent level of detail, LOD(N−1). Note thatattribute signals for the lower level of detail, LOD(N−1) may also bereceived at the second (or subsequent) split operator.

FIG. 13B illustrates an inverse transformation that may be applied at adecoder to decode attribute information of a point cloud, according tosome embodiments.

In some embodiments, a decoder may utilize an inverse transformationprocess as shown in FIG. 13B to reconstruct a point cloud from acompressed point cloud. For example, in some embodiments, performingprediction as described in FIG. 12B at 1258, applying an update operatoras described in FIG. 12B at 1260, applying attribute correction valuesas described in FIG. 12B at 1262 and assigning attributes to points in alevel of detail as described in FIG. 12B at 1264, may be performedaccording to an inverse transformation process as described in FIG. 13B.

In some embodiments, an inverse transformation process may receive anupdated low level resolution sample L′(0) for a lowest level of detailof a LOD structure. The inverse transformation process may also receiveattribute correction values for points not included in the updated lowresolution sample L′(0). For example, for a particular LOD, L′(0) mayinclude a sub-sampling of the points included in the LOD and aprediction technique may be used to determine other points of the LOD,such as would be included in a high resolution sample of the LOD. Theattribute correction values may be received as indicated at 1306, e.g.D(0). At 1318 an update operation may be performed to account for thesmoothing of the attribute correction values performed at the encoder.For example, update operation 1318 may “undo” the update operation thatwas performed at 1314, wherein the update operation performed at 1314was performed to improve compression efficiency by smoothing theattribute values taking into account relative importance of theattribute values. The update operation may be applied to the updated lowresolution sample L′(0) to generate an “un-smoothed” or non-updated lowresolution sample, L(0). The low resolution sample L(0) may be used by aprediction technique at 1320 to determine attribute values of points notincluded in the low resolution sample. The predicted attribute valuesmay be corrected using the attribute correction values, D(0), todetermine attribute values for points of a high resolution sample of theLOD(0). The low resolution sample and the high resolution sample may becombined at merge operator 1322, and a new updated low resolution samplefor a next level of detail L′(1) may be determined. A similar processmay be repeated for the next level of detail LOD(1) as was described forLOD(0). In some embodiments, an encoder as described in FIG. 13A and adecoder as described in FIG. 13B may repeat their respective processesfor N levels of detail of a point cloud.

More detailed example definitions of LODs and methods to determineupdate operations are described below.

In some embodiments, LODs are defined as follows:

-   -   LOD(0)=R(0)    -   LOD(1)=LOD(0) U R(1)    -   LOD(j)=LOD(j−1) U R(j)    -   LOD(N+1)=LOD(N) U R(N)=entire point cloud

In some embodiments, let A be a set of attributes associated with apoint cloud. More precisely, let A(P) be the scalar/vector attributeassociated with the point P of the point cloud. An example of attributewould be color described by RGB values.

Let L(j) be the set of attributes associated with LOD(j) and H(j) thoseassociated with R(j). Based on the definition of level of detailsLOD(j), L(j) and H(j) verify the following properties:

-   -   L(N+1)=A and H(N+1)={ }    -   L(j)=L(j−1) U H(j)    -   L(j) and H(j) are disjointed.

In some embodiments, a split operator, such as split operator 1302,takes as input L(j+1) and generates two outputs: (1) the low resolutionsamples L(j) and (2) the high resolution samples H(j).

In some embodiments, a merge operator, such as merge operator 1322,takes as input L(j) and H(j) and produces L(j+1).

As described in more detail above, a prediction operator may be definedon top of an LOD structure. Let (P(ij))_i be the set points of LOD(j)and (Q(ij))_i those belonging to R(j) and let (A(P(ij)))_i and(A(Q(ij)))_i be the attribute values associated with LOD(j) and R(j),respectively.

In some embodiments, a prediction operator predicts the attribute valueA(Q(i,j)) by using the attribute values of its k nearest neighbors inLOD(j−1), denoted ∇(Q(i,j)):

${{Pred}\left( {Q\left( {i,j} \right)} \right)} = {\sum\limits_{P \in {\nabla{({Q{({i,j})}})}}}{{\alpha \left( {P,{Q\left( {i,j} \right)}} \right)}{A(P)}}}$

where α(P, Q(i,j)) are the interpolation weights. For instance, aninverse distance weighted interpolation strategy may be exploited tocompute the interpolation weights.

The prediction residuals, e.g. attribute correction values, D (Q(i,j))are defined as follows:

D(Q(i,j))=A(Q(i,j))−Pred(Q(i,j))

Note that the prediction hierarchy could be described by an orientedgraph G defined as follows:

-   -   Every point Q in point cloud corresponds to a vertex V(Q) of        graph G.    -   Two vertices of the graph G, V(P) and V(Q), are connected by an        edge E(P, Q), if there exist i and j such that        -   Q=Q(i,j) and        -   P∈∇(Q(i,j))    -   The edge E(Q, P), has weight α(P,Q(i,j)).

In such a prediction strategy as described above, points in lower levelsof detail are more influential since they are used more often forprediction.

Let w(P) be the influence weight associated with a point P. w(P) couldbe defined in various ways.

-   -   Approach 1        -   Two vertices V(P) and V(Q) of G are said to be connected if            there is a path x=(E(1), E(2), . . . , E(s)) of edges of G            that connects them. The weight w(x) of the path x is            defined, as follows:

${w(x)} = {\prod\limits_{s = 1}^{s}\; {\alpha \left( {E(s)} \right)}}$

-   -   -   Let X(P) be the set of paths having P as destination. w(P)            is defined as follows:

w(P)=1+Σ_(x∈X(P))(w(x))²  [EQ. 1]

-   -   -   The previous definition could be interpreted as follows.            Suppose that the attribute A(P) is modified by an amount ∈,            then all the attributes associated with points connected to            P are perturbed. Sum of Squared Errors associated with such            perturbation, denoted SSE(P,∈) is given by:

SSE(P,∈)=w(P)∈²

-   -   Approach 2        -   Computing the influence weights as described previously may            be computationally complex, because all the paths need to be            evaluated. However, since the weights α(E(s)) are usually            normalized to be between 0 and 1, the weight w(x) of a path            x decays rapidly with the number of its edges. Therefore,            long paths could be ignored without significantly impacting            the final influence weight to be computed.        -   Based on the previous property, the definition in [EQ. 1]            may be modified to only consider paths with a limited length            or to discard paths with weights known to be lower that a            user-defined threshold. This threshold could be fixed and            known at both the encoder and decoder, or could be            explicitly signaled at or predefined for different stages of            the encoding process, e.g. once for every frame, LOD, or            even after a certain number of signaled points.    -   Approach 3        -   w(P) could be approximated by the following recursive            procedure:            -   Set w(P)=1 for all points            -   Traverse the points according to the inverse of the                order defined by the LOD structure            -   For every point Q(i,j), update the weights of its                neighbors P∈∇(Q(i,j)) as follows:

w(P)←w(P)+w(Q(i,j),j){α(P,Q(i,j))}^(γ)

-   -   -   -   where γ is a parameter usually set to 1 or 2.

    -   Approach 4        -   w(P) could be approximated by the following recursive            procedure:            -   Set w(P)=1 for all points            -   Traverse the points according to the inverse of the                order defined by the LOD structure            -   For every point Q(i,j), update the weights of its                neighbors P∈∇(Q(i,j)) as follows:

w(P)←w(P)+w(Q(i,j),j)f{α(P,Q(i,j))}

-   -   -   -   where f(x) is some function with resulting values in the                range of [0, 1].

In some embodiments, an update operator, such as update operator 1314 or1318, uses the prediction residuals D(Q(i,j)) to update the attributevalues of LOD(j). The update operator could be defined in differentways, such as:

-   -   Approach 1        -   1. Let Δ(P) be the set of points Q(i,j) such that            P∈∇(Q(i,j)).        -   2. The update operation for P is defined as follows:

${{Update}(P)} = \frac{\Sigma_{Q \in {\Delta {(P)}}}\left\lbrack {\left\{ {\alpha \left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)} \times {D(Q)}} \right\rbrack}{\Sigma_{Q \in {\Delta {(P)}}}\left\lbrack {\left\{ {\alpha \left( {P,Q} \right)} \right\}^{\gamma} \times {w(Q)}} \right\rbrack}$

-   -   where y is a parameter usually set to 1 or 2.    -   Approach 2        -   1. Let Δ(P) be the set of points Q(i,j) such that            P∈∇(Q(i,j)).        -   2. The update operation for P is defined as follows:

${{Update}(P)} = \frac{\Sigma_{Q \in {\Delta {(P)}}}\left\lbrack {g\left\{ {\alpha \left( {P,Q} \right)} \right\} \times {w(Q)}{D(Q)}} \right\rbrack}{\Sigma_{Q \in {\Delta {(P)}}}\left\lbrack {g\left\{ {\alpha \left( {P,Q} \right)} \right\} \times {w(Q)}} \right\rbrack}$

-   -   -   -   where g(x) is some function with resulting values in the                range of [0, 1].

    -   Approach 3        -   Compute Update(P) iteratively as follows:            -   1. Initially set Update(P)=0            -   2. Traverse the points according to the inverse of the                order defined by the LOD structure            -   3. For every point Q(i,j), compute the local updates                (u(1), u(2), . . . , u(k)) associated with its neighbors                ∇(Q(i,j))={P(1), P(2), . . . , P(k)} as the solution to                the following minimization problem:

(u(1),u(2), . . . ,u(k))=argmin{τ_(r=1) ^(k)(u(r))²+(D(Q(i,j))−Σ_(r=1)^(k)α(P(r),Q(i,j))u(k))²}

-   -   -   -   4. Update Update(P(r)):

Update(P(r))←Update(P(r))+u(r)

-   -   Approach 4        -   Compute Update(P) iteratively as follows:            -   1. Initially set Update(P)=0            -   2. Traverse the points according to the inverse of the                order defined by the LOD structure            -   3. For every point Q(i,j), compute the local updates                (u(1), u(2), . . . , u(k)) associated with its neighbors                ∇(Q(i,j))={P(1), P(2), . . . , P(k)} as the solution to                the following minimization problem:

(u(1),u(2), . . . ,u(k))=argmin{h(u(1), . . . ,u(k),D(Q(i,j)))}

-   -   -   -   -   Where h can be any function.

            -   4. Update Update(P(r)):

Update(P(r))←Update(P(r))+u(r)

In some embodiments, when leveraging a lifting scheme as describedabove, a quantization step may be applied to computed waveletcoefficients. Such a process may introduce noise and a quality of areconstructed point cloud may depend on the quantization step chosen.Furthermore, as discussed above, perturbing the attributes of points inlower LODs may have more influence on the quality of the reconstructedpoint cloud than perturbing attributes of points in higher LODs.

In some embodiments, the influence weights computed as described abovemay further be leveraged during the transform process in order to guidethe quantization process. For example, the coefficients associated witha point P may be multiplied with a factor of {w(P)}^(β), where β is aparameter usually set to β=0.5. An inverse scaling process by the samefactor is applied after inverse quantization on the decoder side.

In some embodiments, the values of the parameters could be fixed for theentire point cloud and known at both the encoder and decoder, or couldbe explicitly signaled at or predefined for different stages of theencoding process, e.g. once for every point cloud frame, LOD, or evenafter a certain number of signaled points.

In some embodiments, a hardware-friendly implementation of the liftingscheme described above may leverage a fixed-point representation of theweights and lookup tables for the non-linear operations.

In some embodiments, a lifting scheme as described herein may beleveraged for other applications in addition to compression, such asde-noising/filtering, watermarking, segmentation/detection, as well asvarious other applications

In some embodiments, a decoder may employ a complimentary process asdescribed above to decode a compressed point cloud compressed using anoctree compression technique and binary arithmetic encoder as describedabove.

In some embodiments, the lifting scheme as described above may furtherimplement a bottom-up approach to building the levels of detail (LOD).For example, instead of determining predicted values for points and thenassigning the points to different levels of detail, the predicted valuesmay be determined while determining which points are to be included inwhich level of detail. Also, in some embodiments, residual values may bedetermined by comparing the predicted values to the actual values of theoriginal point could. This too may be performed while determining whichpoints are to be included in which levels of detail. Also, in someembodiments, an approximate nearest neighbor search may be used insteadof an exact nearest neighbor search to accelerate level of detailcreation and prediction calculations. In some embodiments, abinary/arithmetic encoder/decoder may be used to compress/decompressquantized computed wavelet coefficients.

As discussed above, a bottom-up approach may build levels of detail(LODs) and compute predicted attribute values simultaneously. In someembodiments, such an approach may proceed as follows:

-   -   Let (P₃)_(i=1 . . . N) be the set of positions associated with        the point cloud points and let (M_(i))_(i=1 . . . N) be the        Morton codes associated with (P_(i))_(i=1 . . . N). Let D₀ and ρ        be the two user-defined parameters specifying the initial        sampling distance and the distance ratio between LODs,        respectively. A Morton code may be used to represent        multi-dimensional data in one dimension, wherein a “Z-Order        function” is applied to the multidimensional data to result in        the one dimensional representation. Note that ρ>1    -   First the points are sorted according to their associated Morton        codes in an ascending order. Let I be the array of point indexes        ordered according to this process.    -   The algorithm proceeds iteratively. At each iteration k, the        points belonging to the LOD k are extracted and their predictors        are built starting from k=0 until all the points are assigned to        an LOD.    -   The sampling distance D is initialized with D=D₀    -   For each iteration k, where k=0 . . . Number of LODs        -   Let L(k) be the set of indexes of the points belonging to            k-th LOD and O(k) the set of points belonging to LODs higher            than k. L(k) and O(k) are computed as follows.        -   First, O(k) and L(k) are initialized            -   if k=0,L(k)←{ }. Otherwise, L(k)←L(k−1)            -   O(k)←{ }        -   The point indexes stored in the array I are traversed in            order. Each time an index i is selected and its distance            (e.g., Euclidean or other distance) to the most recent SR1            points added to O(k) is computed. SR1 is a user-defined            parameter that controls the accuracy of the nearest neighbor            search. For instance, SR1 could be chosen as 8 or 16 or 64,            etc. The smaller the value of SR1 the lower the            computational complexity and the accuracy of the nearest            neighbor search. The parameter SR1 is included in the bit            stream. If any of the SR1 distances is lower than D, then i            is appended to the array L(k). Otherwise, i is appended to            the array O(k).            -   The parameter SR1 could be changed adaptively based on                the LOD or/and the number of points traversed.            -   In some embodiments, instead of computing an approximate                nearest neighbor, an exact nearest neighbor search                technique may be applied.            -   In some embodiments, the exact and approximate neighbor                search methods could be combined. In particular,                depending on the LOD and/or the number of points in I,                the method could switch between the exact and                approximate search method. Other criteria, may include                the point cloud density, the distance between the                current point and the previous one, or any other                criteria related to the point cloud distribution.        -   This process is iterated until all the indexes in I are            traversed.        -   At this stage, L(k) and O(k) are computed and will be used            in the next steps to build the predictors associated with            the points of L(k).        -   More precisely, let R(k)=L(k)\L(k−1) (where \ is the            difference operator) be the set of points that need to be            added to LOD(k−1) to get LOD(k). For each point i in R(k),            we would like to find the h-nearest neighbors (h is            user-defined parameters that controls the maximum number of            neighbors used for prediction) of i in O(k) and compute the            prediction weights (α_(j)(i))_(j=1 . . . h) associated            with i. The algorithm proceeds as follows.        -   Initialize a counter j=0        -   For each point i in R(k)            -   Let M_(i) be the Morton code associated with i and let                M_(j) be the Morton code associated with j-th element of                the array O(k)            -   While (M_(i)≥M_(j) and j<Size Of (O(k))), incrementing                the counter j by one (j←j+1)            -   Compute the distances of M_(i) to the points associated                with the indexes of O(k) that are in the range [j-SR2,                j+SR2] of the array and keep track of the h-nearest                neighbors (n₁, n₂, . . . , n_(h)) and their associated                squared distances(d_(n) ₁ ²(i), d_(n) ₂ ²(i) . . . ,                d_(n) _(h) ²(i)). SR2 is a user-defined parameter that                controls the accuracy of the nearest neighbor search.                Possible values for SR2 are 8, 16, 32, and 64. The                smaller the value of SR2 the lower the computational                complexity and the accuracy of the nearest neighbor                search. The parameter SR2 is included in the bit stream.                The computation of the prediction weights used for                attribute prediction may be the same as described above.                -   The parameter SR2 could be changed adaptively based                    on the LOD or/and the number of points traversed.                -   In some embodiments, instead of computing an                    approximate nearest neighbor, an exact nearest                    neighbor search technique may be used.                -   In some embodiments, the exact and approximate                    neighbor search methods could be combined. In                    particular, depending on the LOD and/or the number                    of points in I, the method could switch between the                    exact and approximate search method. Other criteria,                    may include the point cloud density, the distance                    between the current point and the previous one, or                    any other criteria related to the point cloud                    distribution.                -   If the distance between the current point and the                    last processed point is lower than a threshold, use                    the neighbors of the last point as an initial guess                    and search around them. The threshold could be                    adaptively chosen based on similar criteria as those                    described above. The threshold could be signaled in                    the bit stream or known to both encoder and decoder.                -   The previous idea could be generalized to n=1,2,3,4                    . . . last points                -   Exclude points with a distance higher that a                    user-defined threshold. The threshold could be                    adaptively chosen based on similar criteria as those                    described above. The threshold could be signaled in                    the bitstream or known to both encoder and decoder.        -   I←O(k)        -   D←D×p        -   The approach described above could be used with any metric            (e.g., L2, L1, Lp) or any approximation of these metrics.            For example, in some embodiments distance comparisons may            use a Euclidean distance comparison approximation, such as a            Taxicab/Manhattan/L1 approximation, or an Octagonal            approximation.

In some embodiments, a lifting scheme may be applied without determininga hierarchy of levels. In such embodiments, the technique may proceed asfollows:

-   -   Sort the input points according to the Morton codes associated        with their coordinates    -   Encode/decode point attributes according to the Morton order    -   For each point i, look for the h nearest neighbors (n₁, n₂, . .        . , n_(h)) already processed (n_(j)<i)    -   Compute the prediction weights as described above.    -   Apply the adaptive scheme described above in order to adjust the        prediction strategy.    -   Predict attributes and entropy encode them as described below.

Binary Arithmetic Coding of Quantized Lifting Coefficients

In some embodiments, lifting scheme coefficients may be non-binaryvalues. In some embodiments, an arithmetic encoder, such as arithmeticencoder 212, that was described above as a component of encoded 202illustrated in FIG. 2B and which used a binary arithmetic codec toencode the 256-value occupancy symbols may also be used to encodelifting scheme coefficients. Or, in some embodiments a similararithmetic encoder may be used. For example, the technique may proceedas follows:

-   -   Mono-dimensional attribute        -   Let C be the quantized coefficient to be encoded. First C is            mapped to a positive number using a function that maps            positive numbers to even numbers and negative numbers to odd            numbers.        -   Let M(C) be the mapped value.        -   A binary value is then encoded to indicate whether C is 0 or            not        -   If C is not zero, then two cases are distinguished            -   If M(C) is higher or equal than alphabetSize (e.g. the                number of symbols supported by the binary arithmetic                encoding technique described above in regard to FIGS.                12C-12F), then the value alphabetSize is encoded by                using the method described above. The difference between                M(C) and alphabetSize is encoded by using an exponential                Golomb coding            -   Otherwise, the value of M(C) is encoded using the method                described above in regard to FIGS. 12C-12F for N.    -   Three-dimensional signal        -   Let C1, C2, C3 be the quantized coefficients to be encoded.            Let K1, and K2 be two indexes for the contexts to be used to            encode the quantized coefficients, C1, C2, and C3.        -   First C1, C2 and C3 are mapped to a positive number as            described above in regard to FIGS. 12C-12F. Let M(C1), M(C2)            and M(C3) be the mapped values of C1, C2, and C3.        -   M(C1) is encoded.        -   M(C2) is encoded while choosing different contexts (i.e.,            binary arithmetic contexts and the binarization context            defined above in regards to FIG. 12C-12F) based on the            condition of whether C1 is zero or not.        -   M(C3) is encoded while choosing different contexts based on            the conditions C1 is zero or not and C2 is zero or not. If            C1 is zero it is known that the value is at least 16. If the            condition C1 is zero use the binary context K1, if the value            is not zero, decrement the value by 1 (it is known that the            value is at least one or more), then check the value is            below the alphabet size, if so encode the value directly.            Otherwise, encode maximum possible value for the alphabet            size. The difference between the maximum possible value for            the alphabet size and the value of M(C3) will then be            encoded using exponential Golomb encoding.    -   Multi-dimensional signal        -   The same approach described above could be generalized to a            d-dimensional signal. Here, the contexts to encode the k-th            coefficient depending on the values of the previous            coefficients (e.g., last 0, 1, 2, 3, . . . , k−1            coefficients).        -   The number of previous coefficients to consider could be            adaptively chosen depending on any of the criteria described            in the previous section for the selection of SR1 and SR2.

Below is a more detailed discussion of how a point cloud transferalgorithm is utilized to minimize distortion between an original pointcloud and a reconstructed point cloud.

The attribute transfer problem could be defined as follows:

-   -   a. Let PC1=(P1(i))_(i∈{1, . . . , N1}) be a point cloud defined        by its geometry (i.e., 3D positions) (X1(i))_(i∈{1, . . . , N1})        and a set of attributes (e.g., RGB color or reflectance)    -   b. Let PC2(P2(j))_(j∈{1, . . . , N2}) be a re-sampled version of        PC1 and let (X2(j))_(j∈{1, . . . , N2}) be its geometry.    -   c. Then compute the set of attribute of        (A2(j))_(j∈{1, . . . , N2}) associated with the point of PC2        such that the texture distortion is minimized.

In order to solve the texture distortion minimization problem using anattribute transfer algorithm:

-   -   Let P_(2→1) (j)∈PC1 be the nearest neighbor of P2(j)∈PC2 in PC1        and A_(2→1)(j) its attribute value.    -   Let P_(1→2) (i)∈PC2 be the nearest neighbor of P1(i)∈PC1 in PC2        and A_(1→2) (i) its attribute value.    -   Let        _(1→2) (j)=(Q(j,h))_(h∈{1, . . . , H(j)})⊆PC2 be the set of        points of PC2 that share the point P1(i)∈PC1 as their nearest        neighbor and (α(j,h))_(h∈{1, . . . , H(j)}) be their attribute        values    -   Let E_(2→1) be the non-symmetric error computed from PC2 to PC1:        -   E_(2→1)=Σ_(j=1) ^(N2)∥A2(j)−A_(2→1)(j)∥²    -   Let E_(1→2) be the non-symmetric error computed from PC1 to PC2:        -   E_(1→2)=Σ_(i=1) ^(N1)∥A1(j)−A_(1→2)(j)∥²    -   Let E be symmetric error that measures the attribute distortion        between PC2 to PC1:        -   E=max (E_(2→1), E_(1→2))

Then determine the set of attributes (A2(j))_(j∈{1, . . . , N2}) asfollows:

a. Initialize E1→0 and E2→0

b. Loop over all the point of PC2

  1) For each point P2(j) compute P_(2→1)(j) ∈ PC1 and 

_(1→2)(j) 2) If (E1 > E2 or 

_(1→2)(j) = {})   A2(j) = A_(2→1)(j) 3) Else   ${A\; 2(j)} = {\frac{1}{H(j)}{\sum\limits_{h = 1}^{H{(j)}}\; {\alpha \left( {j,h} \right)}}}$4) EndIf 5) E1 ← E1 + ∥A2(j) − A_(2→1)(j)∥² 6)$\left. {E\; 2}\leftarrow{{E\; 2} + {{{A\; 2(j)} - {\frac{1}{H(j)}{\sum\limits_{h = 1}^{H{(j)}}\; {\alpha \left( {j,h} \right)}}}}}^{2}} \right.$

Adaptive Attribute Prediction

In some embodiments, an encoder as described above may furtheradaptively change a prediction strategy and/or a number of points usedin a given prediction strategy based on attribute values of neighboringpoints. Also, a decoder may similarly adaptively change a predictionstrategy and/or a number of points used in a given prediction strategybased on reconstructed attribute values of neighboring points.

For example, a point cloud may include points representing a road wherethe road is black with a white stripe on the road. A default nearestneighbor prediction strategy may be adaptively changed to take intoaccount the variability of attribute values for points representing thewhite line and the black road. Because these points have a largedifference in attribute values, a default nearest neighbor predictionstrategy may result in blurring of the white line and/or high residualvalues that decrease a compression efficiency. However, an updatedprediction strategy may account for this variability by selecting abetter suited prediction strategy and/or by using less points in a Knearest neighbor prediction. For example, for the black road, not usingthe white line points in a K nearest neighbor prediction.

In some embodiments, before predicting an attribute value for a point P,an encoder or decoder may compute the variability of attribute values ofpoints in a neighborhood of point P, for example the K-nearestneighboring points. In some embodiments, variability may be computedbased on a variance, a maximum difference between any two attributevalues (or reconstructed attribute values) of the points neighboringpoint P. In some embodiments, variability may be computed based on aweighted average of the neighboring points, wherein the weighted averageaccounts for distances of the neighboring points to point P. In someembodiments, variability for a group of neighboring points may becomputed based on a weighted averages for attributes for the neighboringpoints and taking into account distances to the neighboring points. Forexample,

Variability=E[(X−weighted mean(X))²]

In the above equation, E is the mean value of the distance X, theweighted mean(X) is a weighted mean of the attributes of the points inthe neighborhood of point P that takes into account the distances, X, ofthe neighboring points from point P. In some embodiments, thevariability may be calculated as the maximum difference compared to themean value of the attributes, E(X), the weighted mean of the attributes,weighted mean(X), or the median value of the attributes, median(X). Insome embodiments, the variability may be calculated using the average ofthe values corresponding to the x percent, e.g. x=10 that have thelargest difference as compared to the mean value of the attributes,E(X), the weighted mean of the attributes, weighted mean(X), or themedian value of the attributes, median(X).

In some embodiments, if the calculated variability of the attributes ofthe points in the neighborhood of point P is greater than a thresholdvalue, then a rate-distortion optimization may be applied. For example,a rate-distortion optimization may reduce a number of neighboring pointsused in a prediction or switch to a different prediction technique. Insome embodiments, the threshold may be explicitly written in thebit-stream. Also, in some embodiments, the threshold may be adaptivelyadjusted per point cloud, or sub-block of the point cloud or for anumber of points to be encoded. For example, a threshold may be includedin compressed attribute information file 1150 as configurationinformation 1152, as described below in regard to FIG. 11B.

In some embodiments, different distortion measures may be used in arate-distortion optimization procedure, such as sum of squares error,weighted sum of squares error, sum of absolute differences, or weightedsum of absolute differences.

In some embodiments, distortion could be computed independently for eachattribute, or multiple attributes corresponding to the same sample andcould be considered, and appropriately weighted. For example, distortionvalues for R, G, B or Y, U, V could be computed and then combinedtogether linearly or non-linearly to generate an overall distortionvalue.

In some embodiments, advanced techniques for rate distortionquantization, such as trellis based quantization could also beconsidered where, instead of considering a single point in isolationmultiple points are coded jointly. The coding process, for example, mayselect to encode all these multiple points using the method that resultsin minimizing a cost function of the form J=D+lambda*Rate, where D isthe overall distortion for all these points, and Rate is the overallrate cost for coding these points.

In some embodiments, an encoder, such as encoder 202, may explicitlyencode an index value of a chosen prediction strategy for a point cloud,for a level of detail of a point cloud, or for a group of points withina level of detail of a point cloud, wherein the decoder has access to aninstance of the index and can determine the chosen prediction strategybased on the received index value. The decoder may apply the chosenprediction strategy for the set of points for which the rate-distortionoptimization procedure is being applied. In some embodiments, there maybe a default prediction strategy and the decoder may apply the defaultprediction strategy if no rate-distortion optimization procedure isspecified in the encoded bit stream.

Point Cloud Attribute Transfer Algorithm

In some embodiments, a point cloud transfer algorithm may be used tominimize distortion between an original point cloud and a reconstructedversion of the original point cloud. A transfer algorithm may be used toevaluate distortion due to the original point cloud and thereconstructed point cloud having points that are in slightly differentpositions. For example, a reconstructed point cloud may have a similarshape as an original point cloud, but may have a.) a different number oftotal points and/or b.) points that are slightly shifted as compared toa corresponding point in the original point cloud. In some embodiments,a point cloud transfer algorithm may allow the attribute values for areconstructed point cloud to be selected such that distortion betweenthe original point cloud and a reconstructed version of the originalpoint cloud is minimized. For example, for an original point cloud, boththe positions of the points and the attribute values of the points areknown. However, for a reconstructed point cloud, the position values maybe known (for example based on a sub-sampling process, K-D tree process,or patch image process as described above). However, attribute valuesfor the reconstructed point cloud may still need to be determined.Accordingly a point cloud transfer algorithm can be used to minimizedistortion by selecting attribute values for the reconstructed pointcloud that minimize distortion.

The distortion from the original point cloud to the reconstructed pointcloud can be determined for a selected attribute value. Likewise thedistortion from the reconstructed point cloud to the original pointcloud can be determined for the selected attribute value for thereconstructed point cloud. In many circumstances, these distortions arenot symmetric. The point cloud transfer algorithm is initialized withtwo errors (E21) and (E12), where E21 is the error from the second orreconstructed point cloud to the original or first point cloud and E12is the error from the first or original point cloud to the second orreconstructed point cloud. For each point in the second point cloud, itis determined whether the point should be assigned the attribute valueof the corresponding point in the original point cloud, or an averageattribute value of the nearest neighbors to the corresponding point inthe original point cloud. The attribute value is selected based on thesmallest error.

Exampled Applications for Point Cloud Compression and Decompression

FIG. 14 illustrates compressed point clouds being used in a 3-Dtelepresence application, according to some embodiments.

In some embodiments, a sensor, such as sensor 102, an encoder, such asencoder 104 or encoder 202, and a decoder, such as decoder 116 ordecoder 220, may be used to communicate point clouds in a 3-Dtelepresence application. For example, a sensor, such as sensor 102, at1402 may capture a 3D image and at 1404, the sensor or a processorassociated with the sensor may perform a 3D reconstruction based onsensed data to generate a point cloud.

At 1406, an encoder such as encoder 104 or 202 may compress the pointcloud and at 1408 the encoder or a post processor may packetize andtransmit the compressed point cloud, via a network 1410. At 1412, thepackets may be received at a destination location that includes adecoder, such as decoder 116 or decoder 220. The decoder may decompressthe point cloud at 1414 and the decompressed point cloud may be renderedat 1416. In some embodiments a 3-D telepresence application may transmitpoint cloud data in real time such that a display at 1416 representsimages being observed at 1402. For example, a camera in a canyon mayallow a remote user to experience walking through a virtual canyon at1416

FIG. 15 illustrates compressed point clouds being used in a virtualreality (VR) or augmented reality (AR) application, according to someembodiments.

In some embodiments, point clouds may be generated in software (forexample as opposed to being captured by a sensor). For example, at 1502virtual reality or augmented reality content is produced. The virtualreality or augmented reality content may include point cloud data andnon-point cloud data. For example, a non-point cloud character maytraverse a landscape represented by point clouds, as one example. At1504, the point cloud data may be compressed and at 1506 the compressedpoint cloud data and non-point cloud data may be packetized andtransmitted via a network 1508. For example, the virtual reality oraugmented reality content produced at 1502 may be produced at a remoteserver and communicated to a VR or AR content consumer via network 1508.At 1510, the packets may be received and synchronized at the VR or ARconsumer's device. A decoder operating at the VR or AR consumer's devicemay decompress the compressed point cloud at 1512 and the point cloudand non-point cloud data may be rendered in real time, for example in ahead mounted display of the VR or AR consumer's device. In someembodiments, point cloud data may be generated, compressed,decompressed, and rendered responsive to the VR or AR consumermanipulating the head mounted display to look in different directions.

In some embodiments, point cloud compression as described herein may beused in various other applications, such as geographic informationsystems, sports replay broadcasting, museum displays, autonomousnavigation, etc.

Example Computer System

FIG. 16 illustrates an example computer system 1600 that may implementan encoder or decoder or any other ones of the components describedherein, (e.g., any of the components described above with reference toFIGS. 1-15), in accordance with some embodiments. The computer system1600 may be configured to execute any or all of the embodimentsdescribed above. In different embodiments, computer system 1600 may beany of various types of devices, including, but not limited to, apersonal computer system, desktop computer, laptop, notebook, tablet,slate, pad, or netbook computer, mainframe computer system, handheldcomputer, workstation, network computer, a camera, a set top box, amobile device, a consumer device, video game console, handheld videogame device, application server, storage device, a television, a videorecording device, a peripheral device such as a switch, modem, router,or in general any type of computing or electronic device.

Various embodiments of a point cloud encoder or decoder, as describedherein may be executed in one or more computer systems 1600, which mayinteract with various other devices. Note that any component, action, orfunctionality described above with respect to FIGS. 1-15 may beimplemented on one or more computers configured as computer system 1600of FIG. 16, according to various embodiments. In the illustratedembodiment, computer system 1600 includes one or more processors 1610coupled to a system memory 1620 via an input/output (I/O) interface1630. Computer system 1600 further includes a network interface 1640coupled to I/O interface 1630, and one or more input/output devices1650, such as cursor control device 1660, keyboard 1670, and display(s)1680. In some cases, it is contemplated that embodiments may beimplemented using a single instance of computer system 1600, while inother embodiments multiple such systems, or multiple nodes making upcomputer system 1600, may be configured to host different portions orinstances of embodiments. For example, in one embodiment some elementsmay be implemented via one or more nodes of computer system 1600 thatare distinct from those nodes implementing other elements.

In various embodiments, computer system 1600 may be a uniprocessorsystem including one processor 1610, or a multiprocessor systemincluding several processors 1610 (e.g., two, four, eight, or anothersuitable number). Processors 1610 may be any suitable processor capableof executing instructions. For example, in various embodimentsprocessors 1610 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 1610 may commonly,but not necessarily, implement the same ISA.

System memory 1620 may be configured to store point cloud compression orpoint cloud decompression program instructions 1622 and/or sensor dataaccessible by processor 1610. In various embodiments, system memory 1620may be implemented using any suitable memory technology, such as staticrandom access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions 1622 may be configured toimplement an image sensor control application incorporating any of thefunctionality described above. In some embodiments, program instructionsand/or data may be received, sent or stored upon different types ofcomputer-accessible media or on similar media separate from systemmemory 1620 or computer system 1600. While computer system 1600 isdescribed as implementing the functionality of functional blocks ofprevious Figures, any of the functionality described herein may beimplemented via such a computer system.

In one embodiment, I/O interface 1630 may be configured to coordinateI/O traffic between processor 1610, system memory 1620, and anyperipheral devices in the device, including network interface 1640 orother peripheral interfaces, such as input/output devices 1650. In someembodiments, I/O interface 1630 may perform any necessary protocol,timing or other data transformations to convert data signals from onecomponent (e.g., system memory 1620) into a format suitable for use byanother component (e.g., processor 1610). In some embodiments, I/Ointerface 1630 may include support for devices attached through varioustypes of peripheral buses, such as a variant of the Peripheral ComponentInterconnect (PCI) bus standard or the Universal Serial Bus (USB)standard, for example. In some embodiments, the function of I/Ointerface 1630 may be split into two or more separate components, suchas a north bridge and a south bridge, for example. Also, in someembodiments some or all of the functionality of I/O interface 1630, suchas an interface to system memory 1620, may be incorporated directly intoprocessor 1610.

Network interface 1640 may be configured to allow data to be exchangedbetween computer system 1600 and other devices attached to a network1685 (e.g., carrier or agent devices) or between nodes of computersystem 1600. Network 1685 may in various embodiments include one or morenetworks including but not limited to Local Area Networks (LANs) (e.g.,an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., theInternet), wireless data networks, some other electronic data network,or some combination thereof. In various embodiments, network interface1640 may support communication via wired or wireless general datanetworks, such as any suitable type of Ethernet network, for example;via telecommunications/telephony networks such as analog voice networksor digital fiber communications networks; via storage area networks suchas Fibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices 1650 may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, or any other devices suitable forentering or accessing data by one or more computer systems 1600.Multiple input/output devices 1650 may be present in computer system1600 or may be distributed on various nodes of computer system 1600. Insome embodiments, similar input/output devices may be separate fromcomputer system 1600 and may interact with one or more nodes of computersystem 1600 through a wired or wireless connection, such as over networkinterface 1640.

As shown in FIG. 16, memory 1620 may include program instructions 1622,which may be processor-executable to implement any element or actiondescribed above. In one embodiment, the program instructions mayimplement the methods described above. In other embodiments, differentelements and data may be included. Note that data may include any dataor information described above.

Those skilled in the art will appreciate that computer system 1600 ismerely illustrative and is not intended to limit the scope ofembodiments. In particular, the computer system and devices may includeany combination of hardware or software that can perform the indicatedfunctions, including computers, network devices, Internet appliances,PDAs, wireless phones, pagers, etc. Computer system 1600 may also beconnected to other devices that are not illustrated, or instead mayoperate as a stand-alone system. In addition, the functionality providedby the illustrated components may in some embodiments be combined infewer components or distributed in additional components. Similarly, insome embodiments, the functionality of some of the illustratedcomponents may not be provided and/or other additional functionality maybe available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 1600 may be transmitted to computer system1600 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link. Various embodiments mayfurther include receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-accessible medium. Generally speaking, a computer-accessiblemedium may include a non-transitory, computer-readable storage medium ormemory medium such as magnetic or optical media, e.g., disk orDVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR,RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessiblemedium may include transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

Various embodiments can be described in view of the following clauses:

1. A system comprising:

-   -   one or more sensors configured to capture a plurality of points        that make up a point cloud, wherein respective ones of the        points comprises spatial information for the point and attribute        information for the point; and    -   an encoder configured to compress the attribute information for        the points, wherein to compress the attribute information, the        encoder is configured to:        -   assign an attribute value to at least one point of the point            cloud based, at least in part, on the attribute information            included in the captured point cloud for the point; and        -   for each of respective other ones of the points of the point            cloud:            -   identify a set of neighboring points;            -   determine a predicted attribute value for the respective                point based, at least in part, on predicted or assigned                attributes values for the neighboring points; and            -   determine, based, at least in part, on comparing the                predicted attribute value for the respective point to                the attribute information for the point included in the                captured point cloud, an attribute correction value for                the point; and        -   encode the compressed attribute information, wherein the            compressed attribute information comprises:            -   the assigned attribute value for the at least one point;                and data indicating, for the respective other ones of                the points, the determined attribute correction values.

2. The system of clause 1, wherein to determine, for each of therespective other ones of the points, the predicted attribute value, theencoder is further configured to:

-   -   determine, for each of the respective other ones of the points,        the predicted attribute value based, at least in part, on        respective distances between the point and respective ones of        the neighboring points, wherein attribute values of neighboring        points closer to the point are weighted more heavily than        attribute values of neighboring points that are further away        from the point.

3. The system of clause 2, wherein the encoder is further configured to:

-   -   determine a minimum spanning tree for the points of the point        cloud,    -   wherein to identify the set of neighboring points, the encoder        is configured to identify a set of nearest neighboring points        according to the minimum spanning tree.

4. The system of clause 3, wherein the encoder is configured to:

-   -   evaluate the respective other ones of the points of the point        cloud according to an order determined based, at least in part,        on the minimum spanning tree; and    -   encode the determined attribute correction values for the        respective other ones of the points according to the order.

5. The system of clause 1, wherein the encoder is configured to encodethe data indicating, for the respective other ones of the points, theattribute correction value according to an arithmetic compressionalgorithm.

6. The system of clause 1, wherein the encoder is configured to encodethe data indicating, for the respective other ones of the points, theattribute correction value according to a Golomb compression algorithm.

7. The system of clause 1, wherein the encoder stores a plurality ofattribute correction value encoding contexts, wherein different ones ofthe attribute correction value encoding contexts are selected forencoding an attribute correction value based, at least in part, on anumber of symbols included in the attribute correction value.

8. A method of compressing attribute information for a point cloudcomprising:

-   -   assigning an attribute value to at least one point of the point        cloud based, at least in part, on attribute information for the        at least one point included in the point cloud, wherein the        point cloud comprises spatial information for a plurality of        points and attribute information specifying one or more        attributes for respective ones of the plurality of points; and    -   for each of respective other ones of the points of the point        cloud:        -   identifying a set of neighboring points;        -   determining a predicted attribute value for the point based,            at least in part, on predicted or assigned attribute values            for the neighboring points; and        -   determining, based, at least in part, on comparing the            predicted attribute value for the point to the attribute            information for the point, an attribute correction value for            the point; and    -   encoding compressed attribute information for the point cloud        comprising:        -   the assigned attribute value for the at least one point; and        -   data indicating, for the respective other ones of the            points, the determined attribute correction values.

9. The method of clause 8, wherein a number of neighboring points toidentify, for the set of nearest neighboring points, is a configurableparameter, and

-   -   wherein said encoding the compressed attribute information for        the compressed point cloud further comprises encoding        configuration information indicating the number of nearest        neighboring points to include in the set of identified nearest        neighboring points.

10. The method of clause 9, wherein said determining the predictedattribute value for the point comprises:

-   -   determining respective distances between the point and        respective ones of the neighboring points of the set of nearest        neighboring points,    -   wherein the attribute value for the point is determined based,        at least in part, on an inverse distance interpolation method,        wherein attribute values of neighboring points closer to the        point are weighted more heavily than attribute values of        neighboring points that are further away from the point.

11. The method of clause 10, further comprising:

-   -   determining a minimum spanning tree for the points of the point        cloud,    -   wherein said identifying the set of neighboring points comprises        identifying a set of nearest neighboring points according to the        minimum spanning tree.

12. The method of clause 11, wherein for the respective other ones ofthe points of the point cloud, the respective points are evaluatedaccording to a processing order determined based, at least in part, onthe minimum spanning tree, wherein the minimum spanning tree istraversed according to minimum distances between successive points beingevaluated.

13. The method of clause 11, further comprising:

-   -   encoding a K-D tree to compress spatial information of the point        cloud.

14. The method of clause 13, wherein at least some of the points of thepoint cloud comprise attribute information specifying more than oneattribute, wherein encoding the compressed attribute information furthercomprises:

-   -   encoding data indicating an attribute correction value for a        first attribute of the point according to an encoding context        selected based, at least in part, on a number of symbols        included in the attribute correction value for the first        attribute; and    -   encoding data indicating one or more additional attributes of        the point according to the selected encoding context.

15. The method of clause 8, wherein said encoding compressed attributeinformation comprises:

-   -   encoding positive values as either even or odd numbers; and    -   encoding negative values as either even or odd numbers,        wherein positive and negative values are not both encoded as        even values or odd values.

16. A non-transitory computer-readable medium storing programinstructions that, when executed by one or more processors, cause theone or more processors to implement an encoder configured to:

-   -   assign an attribute value to at least one point of the point        cloud based, at least in part, on attribute information for the        at least one point included in the point cloud, wherein the        point cloud comprises spatial information for a plurality of        points and attribute information specifying one or more        attributes for respective ones of the plurality of points; and    -   for each of respective other ones of the points of the point        cloud:        -   identify a set of neighboring points;        -   determine a predicted attribute value for the point based,            at least in part, on predicted or assigned attribute values            for the neighboring points; and        -   determine, based, at least in part, on comparing the            predicted attribute value for the point to the attribute            information for the point, an attribute correction value for            the point; and    -   encode compressed attribute information for the point cloud        comprising:        -   the assigned attribute value for the at least one point; and        -   data indicating, for the respective other ones of the            points, the determined attribute correction values.

17. The non-transitory computer-readable medium of clause 16, wherein anumber of neighboring points to identify, for the set of nearestneighboring points, is a configurable parameter, and

wherein said encode the compressed attribute information for thecompressed point cloud further comprises encode configurationinformation indicating the number of nearest neighboring points toinclude in the set of identified nearest neighboring points.

18. The non-transitory computer-readable medium of clause 17, whereinsaid determine the predicted attribute value for the point comprises:

-   -   determine respective distances between the point and respective        ones of the neighboring points of the set of nearest neighboring        points,    -   wherein the attribute value for the point is determined based,        at least in part, on an inverse distance interpolation method,        wherein attribute values of neighboring points closer to the        point are weighted more heavily than attribute values of        neighboring points that are further away from the point.

19. The non-transitory computer-readable medium of clause 18, whereinthe encoder is further configured to:

-   -   determine a minimum spanning tree for the points of the point        cloud,    -   wherein said identify the set of neighboring points comprises        identifying a set of nearest neighboring points according to the        minimum spanning tree.

20. The non-transitory computer-readable medium of clause 19, whereinfor the respective other ones of the points of the point cloud, therespective points are evaluated according to a processing orderdetermined based, at least in part, on the minimum spanning tree,wherein the minimum spanning tree is traversed according to minimumdistances between successive points being evaluated.

21. A system comprising:

-   -   one or more sensors configured to capture a plurality of points        that make up a point cloud, wherein respective ones of the        points comprise spatial information for the point and attribute        information for the point; and    -   an encoder configured to:        -   assign an attribute value to at least one point of the point            cloud based on the attribute information included in the            captured point cloud for the point; and        -   for a sub-set of respective ones of the points of the point            cloud not included in a previously determined level of            detail:            -   determine a predicted attribute value for the respective                point based on predicted or assigned attributes values                for neighboring points; and            -   determine, based on comparing the predicted attribute                value for the respective point to the attribute                information for the point included in the captured point                cloud, an attribute correction value for the point;        -   apply an update operation to smooth the determined attribute            correction values, wherein the update operation takes into            account relative influence of the attributes of the            respective points on other levels of detail; and        -   encode the assigned attribute value and the updated            attribute correction values for first level of detail and            the one or more additional levels of detail.

22. A decoder configured to:

-   -   receive compressed attribute information for a point cloud        comprising at least one assigned attribute value for at least        one point of the point cloud and data indicating attribute        correction values for attributes of other points of the point        cloud; and    -   provide attribute information for a decompressed point cloud        having a first level of detail, wherein said providing attribute        information comprises performing an update operation to remove        attribute value smoothing applied an encoder; and    -   update the decompressed point cloud to include attribute        information for additional sub-sets of points at one or more        other ones of the plurality of levels of detail.

23. A method comprising:

-   -   receiving compressed attribute information for a point cloud        comprising at least one assigned attribute value for at least        one point of the point cloud and data indicating attribute        correction values for attributes of the other points of the        point cloud; and    -   providing attribute information for a decompressed point cloud        having a first level of detail, wherein said providing attribute        information comprises performing an update operation to remove        attribute value smoothing applied an encoder; and    -   updating the decompressed point cloud to include attribute        information for additional sub-sets of points at one or more        other ones of the plurality of levels of detail.

24. A non-transitory computer readable medium storing programinstructions, that when executed by one or more processors, cause theone or more processors to perform:

-   -   receiving compressed attribute information for a point cloud        comprising at least one assigned attribute value for at least        one point of the point cloud and data indicating attribute        correction values for attributes of the other points of the        point cloud; and    -   providing attribute information for a decompressed point cloud        having a first level of detail, wherein said providing attribute        information comprises performing an update operation to remove        attribute value smoothing applied an encoder; and    -   updating the decompressed point cloud to include attribute        information for additional sub-sets of points at one or more        other ones of the plurality of levels of detail.

25. A method comprising:

-   -   assigning an attribute value to at least one point of a point        cloud based on attribute information included in a captured        point cloud for the point; and    -   for multiple points of a sub-set of respective points of the        point cloud not included in a previously determined level of        detail:        -   determining a predicted attribute value for respective            points of the sub-set of points based on predicted or            assigned attributes values for neighboring points;        -   determining respective attribute correction values for the            multiple points, based on comparing the predicted attribute            values for the respective points to the attribute            information for the points included in the captured point            cloud;        -   applying an update operation to smooth the determined            attribute correction values, wherein the update operation            takes into account relative influence of the attributes of            the respective points on other levels of detail; and        -   encoding the assigned attribute value and the updated            attribute correction values for first level of detail and            the one or more additional levels of detail.

26. A non-transitory computer readable medium storing programinstructions, that when executed by one or more processors, cause theone or more processors to perform:

-   -   assigning an attribute value to at least one point of a point        cloud based on attribute information included in a captured        point cloud for the point; and    -   for multiple points of a sub-set of respective points of the        point cloud not included in a previously determined level of        detail:        -   determining a predicted attribute value for respective            points of the sub-set of points based on predicted or            assigned attributes values for neighboring points;        -   determining respective attribute correction values for the            multiple points, based on comparing the predicted attribute            values for the respective points to the attribute            information for the points included in the captured point            cloud;        -   applying an update operation to smooth the determined            attribute correction values, wherein the update operation            takes into account relative influence of the attributes of            the respective points on other levels of detail; and        -   encoding the assigned attribute value and the updated            attribute correction values for first level of detail and            the one or more additional levels of detail.

27. A system comprising:

-   -   one or more sensors configured to capture a plurality of points        that make up a point cloud, wherein respective ones of the        points comprise spatial information for the point and attribute        information for the point; and    -   an encoder configured to:        -   determine predicted attribute values for respective ones of            the points based on predicted or assigned attributes values            for neighboring points, wherein the determined predicted            attribute values are determined prior to or while level of            detail assignments are being determined for the respective            points and the neighboring points; and        -   determine, based on comparing the predicted attribute values            for the respective points to the attribute information for            the respective points included in the captured point cloud,            respective attribute correction values for the respective            points.

28. A system comprising:

-   -   one or more sensors configured to capture a plurality of points        that make up a point cloud, wherein respective ones of the        points comprise spatial information for the point and attribute        information for the point; and    -   an encoder configured to:        -   for a sub-set of respective ones of the points of the point            cloud:            -   identify a set of neighboring points greater than a                first distance from the point;            -   determine whether a variability of the neighboring                points exceeds a variability threshold;            -   determine respective predicted attribute values for one                or more points of the sub-set of respective points                according to a first prediction procedure if the                variability of the neighboring points is less than the                threshold and determine the respective predicted                attribute values for the one or more point according to                another prediction procedure if the variability of the                neighboring points exceeds the threshold; and            -   determine, based on comparing the respective predicted                attribute values for the respective ones of the one or                more points to the attribute information for                corresponding respective points included in the captured                point cloud, respective attribute correction values for                the one or more points; and        -   encode the assigned attribute value and the determined            attribute correction values.

29. A decoder configured to:

-   -   receive compressed attribute information for a point cloud        comprising data indicating attribute correction values for        attributes of points of the point cloud and indication of a        selected prediction strategy to be applied to decompress the        compressed attribute information, wherein the selected        prediction strategy is one of a plurality of prediction        strategies supported by the decoder;    -   predict attribute values for points of the point cloud based on        the selected prediction strategy;    -   apply one or more attribute correction values to the predicted        attribute values to determine attribute information for a        decompressed point cloud.

30. A method comprising:

-   -   receiving compressed attribute information for a point cloud        comprising data indicating attribute correction values for        attributes of points of the point cloud and an indication of a        selected prediction strategy to be applied to decompress the        compressed attribute information, wherein the selected        prediction strategy is one of a plurality of prediction        strategies supported by a decoder;    -   predicting attribute values for points of the point cloud based        on the selected prediction strategy;    -   applying one or more attribute correction values to the        predicted attribute values to determine attribute information        for a decompressed point cloud.

31. A method comprising:

-   -   identifying a set of neighboring points greater than a first        distance from a point of a point cloud;    -   determining whether a variability of the neighboring points        exceeds a variability threshold;    -   determining respective predicted attribute values for one or        more points of the sub-set of respective points according to a        first prediction procedure if the variability of the neighboring        points is less than the threshold and determine the respective        predicted attribute values for the one or more point according        to another prediction procedure if the variability of the        neighboring points exceeds the threshold;    -   determining, based on comparing the respective predicted        attribute values for the respective ones of the one or more        points to the attribute information for corresponding respective        points included in the captured point cloud, respective        attribute correction values for the one or more points; and        encoding the assigned attribute value and the determined        attribute correction values.

32 A non-transitory computer-readable medium storing programinstructions, that when executed by one or more processors, causes theone or more processors to perform:

-   -   identifying a set of neighboring points greater than a first        distance from a point of a point cloud;    -   determining whether a variability of the neighboring points        exceeds a variability threshold;    -   determining respective predicted attribute values for one or        more points of the sub-set of respective points according to a        first prediction procedure if the variability of the neighboring        points is less than the threshold and determine the respective        predicted attribute values for the one or more point according        to another prediction procedure if the variability of the        neighboring points exceeds the threshold;    -   determining, based on comparing the respective predicted        attribute values for the respective ones of the one or more        points to the attribute information for corresponding respective        points included in the captured point cloud, respective        attribute correction values for the one or more points; and        encoding the assigned attribute value and the determined        attribute correction values.

33. A non-transitory computer-readable medium storing programinstructions, that when executed by one or more processors, causes theone or more processors to perform:

-   -   receiving compressed attribute information for a point cloud        comprising data indicating attribute correction values for        attributes of points of the point cloud and an indication of a        selected prediction strategy to be applied to decompress the        compressed attribute information, wherein the selected        prediction strategy is one of a plurality of prediction        strategies supported by a decoder;    -   predicting attribute values for points of the point cloud based        on the selected prediction strategy;    -   applying one or more attribute correction values to the        predicted attribute values to determine attribute information        for a decompressed point cloud.

34. A system comprising:

-   -   a decoder configured to:        -   receive compressed attribute information for a point cloud            comprising at least one assigned attribute value for at            least one point of the point cloud and data indicating, for            other points of the point cloud, respective attribute            correction values for respective attributes of the other            points; and    -   for each of respective other ones of the points of the point        cloud other than the at least one point:        -   identify a set of neighboring points to a point being            evaluated;        -   determine a predicted attribute value for the point being            evaluated based, at least in part, on predicted or assigned            attribute values for the neighboring points; and        -   adjust the predicted attribute value for the point being            evaluated based, at least in part, on an attribute            correction value for the point included in the compressed            attribute information; and        -   provide attribute information for a decompressed point cloud            comprising the at least one assigned attribute value for the            at least one point and the adjusted predicted attribute            values for the other ones of the points.

35. The system of clause 34, wherein the decoder is further configuredto:

-   -   determine a minimum spanning tree for the point cloud based, at        least in part, on spatial information for the points of the        point cloud; and    -   determine a processing order for evaluating the respective other        ones of the points based, at least in part, on the minimum        spanning tree, wherein the processing order is determined based,        at least in part, on minimum distances between successive points        being evaluated

36. The system of clause 35, wherein to identify the set of neighboringpoints, the decoder is configured to identify a set of nearestneighboring points according to the minimum spanning tree.

37. The system of clause 36, wherein the decoder is configured todetermine the attribute predictions based, at least in part, on inversedistance relationships between the point being evaluated and theneighboring points.

38. The system of clause 37, wherein for points comprising more than oneattribute, the decoder is configured to apply the same context used fordecompressing a compressed attribute correction value for a firstpredicted attribute value of a given point when decompressing compressedattribute correction values for one or more other attributes of thegiven point.

39. A method of decompressing attribute information for a point cloudcomprising:

-   -   receiving compressed attribute information for a point cloud        comprising at least one assigned attribute value for at least        one point of the point cloud and data indicating, for other        points of the point cloud, respective attribute correction        values for respective attributes of the other points; and    -   for each of respective other ones of the points of the point        cloud other than the at least one point:        -   identifying a set of neighboring points to a point being            evaluated;        -   determining a predicted attribute value for the point being            evaluated based, at least in part, on predicted or assigned            attribute values for the neighboring points; and        -   adjusting the predicted attribute value for the point being            evaluated based, at least in part, on an attribute            correction value for the point included in the compressed            attribute information; and        -   providing attribute information for a decompressed point            cloud comprising the at least one assigned attribute value            for the at least one point and the adjusted predicted            attribute values for the other ones of the points.

40. The method of clause 39, further comprising:

-   -   determining a minimum spanning tree for the point cloud based,        at least in part, on spatial information for the points of the        point cloud; and    -   determining a processing order for evaluating the respective        other ones of the points based, at least in part, on the minimum        spanning tree, wherein the processing order is determined based,        at least in part, on minimum distances between successive points        being evaluated.

41. The method of clause 40, further comprising:

-   -   identifying a set of nearest neighboring points according to the        minimum spanning tree.

42. The method of clause 41, wherein the attribute predictions aredetermined based, at least in part, on inverse distance relationshipsbetween the point being evaluated and the neighboring points.

43. The method of clause 42, wherein for points comprising more than oneattribute, the same context used for decompressing a compressedattribute correction value for a first predicted attribute value of agiven point is used when decompressing compressed attribute correctionvalues for one or more other attributes of the given point.

44. A non-transitory computer-readable medium storing programinstructions that, when executed by one or more processors, cause theone or more processors to implement a decoder configured to:

-   -   receive compressed attribute information for a point cloud        comprising at least one assigned attribute value for at least        one point of the point cloud and data indicating, for other        points of the point cloud, respective attribute correction        values for respective attributes of the other points; and    -   for each of respective other ones of the points of the point        cloud other than the at least one point:        -   identify a set of neighboring points to a point being            evaluated;        -   determine a predicted attribute value for the point being            evaluated based, at least in part, on predicted or assigned            attribute values for the neighboring points; and        -   adjust the predicted attribute value for the point being            evaluated based, at least in part, on an attribute            correction value for the point included in the compressed            attribute information; and    -   provide attribute information for a decompressed point cloud        comprising the at least one assigned attribute value for the at        least one point and the adjusted predicted attribute values for        the other ones of the points.

45. The non-transitory computer-readable medium of clause 44, whereinthe decoder is further configured to:

-   -   determine a minimum spanning tree for the point cloud based, at        least in part, on spatial information for the points of the        point cloud; and    -   determine a processing order for evaluating the respective other        ones of the points based, at least in part, on the minimum        spanning tree, wherein the processing order is determined based,        at least in part, on minimum distances between successive points        being evaluated.

46. The non-transitory computer-readable medium of clause 45, wherein toidentify the set of neighboring points, the decoder is configured toidentify a set of nearest neighboring points according to the minimumspanning tree.

47. The non-transitory computer-readable medium of clause 46, whereinthe attribute predictions are determined based, at least in part, oninverse distance relationships between the point being evaluated and theneighboring points.

48. The non-transitory computer-readable medium of clause 47, whereinfor points comprising more than one attribute, the decoder is configuredto apply the same context used for decompressing a compressed attributecorrection value for a first predicted attribute value of a given pointwhen decompressing compressed attribute correction values for one ormore other attributes of the given point.

What is claimed is:
 1. A system comprising: one or more sensorsconfigured to capture a plurality of points that make up a point cloud,wherein respective ones of the points comprise spatial information forthe point and attribute information for the point; and an encoderconfigured to: determine a first level of detail for the attributeinformation of the point cloud; and determine one or more additionallevels of detail for the attribute information of the point cloud,wherein to determine the first level of detail or the one or moreadditional levels of detail, the encoder is configured to: assign anattribute value to at least one point of the point cloud based on theattribute information included in the captured point cloud for thepoint; and for a sub-set of respective ones of the points of the pointcloud not included in a previously determined level of detail: identifya set of neighboring points greater than a first distance from thepoint; determine a predicted attribute value for the respective pointbased on predicted or assigned attributes values for the neighboringpoints; and determine, based on comparing the predicted attribute valuefor the respective point to the attribute information for the pointincluded in the captured point cloud, an attribute correction value forthe point; and encode the assigned attribute value and the determinedattribute correction values for first level of detail and the one ormore additional levels of detail.
 2. The system of claim 1, wherein theencoder is configured to: sequentially provide the first level of detailand the one or more additional levels of detail to a recipient device.3. The system of claim 2, wherein the encoder further encodes thespatial information for the point cloud as a K-D tree.
 4. The system ofclaim 2, wherein the encoder is configured to include spatialinformation for respective sub-sets of points in the respective levelsof detail corresponding to the respective sub-sets of points.
 5. Thesystem of claim 1, wherein the encoder is further figured to encode:information indicating a number of levels of detail encoded for thepoint cloud.
 6. The system of claim 1, wherein the encoder is furtherconfigured to encode: an initial sampling distance for identifyingpoints included in the first level of detail; and a sampling distanceupdate factor for determining additional sampling distances for the oneor more additional levels of detail, wherein the additional samplingdistances are for identifying points included in the one or moreadditional levels of detail, wherein the initial sampling distance andthe sampling distance update factor are provided to a decoder inaddition to the encoded assigned attribute value and the encodeddetermined attribute correction values for first level of detail and theone or more additional levels of detail.
 7. The system of claim 1,wherein, to determine the predicted attribute value for the respectivepoint based on predicted or assigned attributes values for theneighboring points, the encoder is configured to: determine respectivedistances between the respective point and respective ones of theneighboring points of a set of neighboring points, wherein the attributevalue for the respective point is determined based on an inversedistance interpolation method, wherein attribute values of neighboringpoints closer to the respective point are weighted more heavily thanattribute values of neighboring points that are further away from therespective point.
 8. A decoder configured to: receive compressedattribute information for a point cloud comprising at least one assignedattribute value for at least one point of the point cloud and dataindicating attribute correction values for attributes of other points ofthe point cloud, wherein the attribute correction values are ordered ina plurality of levels of detail for a plurality of sub-sets of the otherpoints of the point cloud; and provide attribute information for adecompressed point cloud having a first level of detail; and update thedecompressed point cloud to include attribute information for additionalsub-sets of points at one or more other ones of the plurality of levelsof detail.
 9. The decoder of claim 8, wherein to update the decompressedpoint cloud, the decoder is configured to assign attribute informationto the additional sub-sets of points, wherein the assigned attributeinformation is in addition to attribute information previously assignedfor sub-sets of points included in other ones of the levels of detail.10. The decoder of claim 8, wherein to determine attribute informationfor a sub-set of the points included in the first level of detail or theadditional sub-sets of points included in the one or more other ones ofthe levels of detail, the decoder is configured to: for each of thepoints of a given sub-set of points corresponding to a given level ofdetail: identify a set of neighboring points to a point being evaluated;determine a predicted attribute value for the point being evaluatedbased on predicted or assigned attribute values for the neighboringpoints; and adjust the predicted attribute value for the point beingevaluated based on an attribute correction value for the point includedin the compressed attribute information.
 11. The decoder of claim 8,wherein the predicted attribute values are determined based on inversedistance relationships between the point being evaluated and theneighboring points.
 12. The decoder of claim 8, wherein the decoder isconfigured to determine a number of levels of detail to decode based, atleast in part, on a data budget allocated for the decompressed pointcloud.
 13. The decoder of claim 8, wherein the decoder is configured todetermine a number of levels of detail to decode based, at least inpart, on a viewing mode used to view the decompressed point cloud.
 14. Amethod comprising: receiving compressed attribute information for apoint cloud comprising at least one assigned attribute value for atleast one point of the point cloud and data indicating attributecorrection values for attributes of the other points of the point cloud,wherein the attribute correction values are ordered in a plurality oflevels of detail for a plurality of sub-sets of the other points of thepoint cloud; and providing attribute information for a decompressedpoint cloud having a first level of detail; and updating thedecompressed point cloud to include attribute information for additionalsub-sets of points at one or more other ones of the plurality of levelsof detail.
 15. The method of claim 14, wherein providing the attributeinformation for the first level of detail and updating the decompressedpoint cloud to include attribute information for additional sub-sets ofpoints at one or more other ones of the plurality of levels of detailrespectively comprises: for each of the points of a given sub-set ofpoints corresponding to a given level of detail: identifying a set ofneighboring points to a point being evaluated; determining a predictedattribute value for the point being evaluated based on predicted orassigned attribute values for the neighboring points; and adjusting thepredicted attribute value for the point being evaluated based on anattribute correction value for the point included in the compressedattribute information.
 16. The method of claim 15, further comprising:receiving spatial information for points of the first level of detail;and separately receiving spatial information for points of the secondlevel of detail.
 17. The method of claim 15, further comprising:receiving encoded spatial information for points of more than one levelof detail of the point cloud; and decoding the encoded spatialinformation prior to said providing attribute information for the firstlevel of detail.
 18. The method of claim 15, wherein said updating thedecompressed point cloud comprises: assigning attribute information tothe additional sub-sets of points, wherein the assigned attributeinformation is in addition to attribute information previously assignedfor sub-sets of points included in other ones of the levels of detail.19. The method of claim 15, further comprising: determining a number oflevels of detail to decode based, at least in part, on a data budgetallocated for the decompressed point cloud.
 20. The method of claim 15,further comprising: determining a number of levels of detail to decodebased, at least in part, on a viewing mode used to view the decompressedpoint cloud.